
LUDW 001/01 US 



10 



15 



Processing and Generation of 
Control Signals for Real-Time 
Control of Music Signal Processing, 
Mixing, Video, and Lighting 

BACKGROUND OF THE INVENTION 

Related Cases 

This application is a divisional continuation of U.S. application number 09/313,533, filed 
May 15, 1999, in turn based on U.S. Provisional Serial Number 60/085,713 filed May 15, 
1998. 

O Field of Invention 

This invention relates to musical instrument performance systems and environments, and 
|I in particular to the combination of novel instrument entities built from synergistic arrange - 

ty ments of traditional and novel instrument elements, and the interconnection of said instru- 

cts ment entities utilizing generalized interface entities to signal routing, processing, and 

13 synthesis entities built from synergistic combinations of traditional and novel architectures, 

2§ processes, and methodologies. 

The systems and methods herein are intended to make possible a new generation of musical 
=y instrument products with enhanced capabilities and sounds, new semiotic-oriented perfor- 

mance capabilities, and rich composition and recording environments. 

s 

3 ~ Background 

There has been considerable advancement in music technology in the last several decades, 
35 but recent innovations driven by mass-market forces have narrowed the range of possibil- 

ities for commercially available instruments and the ways in which new recorded and per- 
formed music are being explored. Audio samples of diverse instruments, advanced signal 
processing power, improved fidelity, the MIDI control interface, sequencers, and music 
workstations are important assets but, together with the ways synthesizers, signal process- 
40 ing systems, and instrument controllers have come to be designed, the channel of innova- 

tion is focused on a relatively narrow conceptual range that will consume as much rework 
and refinement energy as can be allotted. A few modern outlier innovations have appeared, 
such as the Roland COSM signal processing methods, Yahama VL1 model-based synthesis 
methods, and Buchla's and Starr Switch alternative MIDI controllers, but due to the fo- 
45 cused drive of the mainstream these exceptions are largely orphaned in their application. 
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What is needed is some reach into the souls (rather than make samples) of deep non- West- 
ern and Western instruments, a recasting of the now institutionalized signal processing 
chains, adaptations of new classes of applicable physical phenomenon, extensions as to the 
types and forms of meaningful human control, and, in the context of performance, a deeper 
5 integration of visual and audio environments. 
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Based on research and development of this nature, it is possible to create a new-generation 
framework for expanding the timbral, expressive range, artistic depth, and semiotic aspects 
of performed and recorded music as well as wide ranges of performance art. Such a frame- 
j 5 work is particularly advantageous if it were to build on and inter- work with both the exist- 

ing music technology mainstream and the long established playing techniques of 
expressively sophisticated, iconic, or significantly adaptable instruments. With such at- 
= tributes, isolated products and musical directions can be gently folded in to the established 

-~ mainstream and evolve as the mainstream finds moments of stagnation and boredom within 

2§ itself. This methodology would permit the current manufacturing and marketing establish- 

jT ments of music technology and content to progressively and profitably shift to a more cre- 

hj atively satisfying and sustainable path. 

*fr To these ends, the invention provides methods, apparatus, and example implementations 

subscribing to a standardized framework which address these needs and opportunities. 

1. A key aspect of the invention is a unified architecture involving instrument 

(3 entities, generalized instrument interfaces, and signal routing, processing, and 

\£ synthesis elements. 

[g 2. A further aspect of the invention is the defining of general instrument elements 

|3 which instrument entities can be created from. 

i= 3. A further aspect of the invention is augmenting existing instruments lending 

themselves to expansion with said general instrument elements. 

4. A further aspect of the invention is the use of miniature keyboards for the 
35 attachment to existing instruments. 

5. A further aspect of the invention is the expansion of keyboards to include any one 
or more of proximate, superimposed, programmable tactical feedback, and/or 
multiple (more than 2) parameter key features. 

40 6. A further aspect of the invention is the sharing of same electronics across multiple 

keyboards and/or strum-pads. 

7. A further aspect of the invention is that of strum-pads with non-repeating contacts 
along the strum path and flexible assignment of note event control signals to each 
contact. 

45 
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8. A further aspect of the invention is that of including standardized arrangements 
of panel controls, such as switches and sliders, to instruments. 

9. A further aspect of the invention is the use of null/contact touch-pads, potentially 
fitted with impact and/or pressure sensors and with the potential derivation of 
multiple contact point information, as a musical interface. 

10. A further aspect of the invention is that of pressure-sensor array touch-pads as an 
instrument controller, potentially including image recognition capabilities and the 
ability to derive and assign control parameters from the way the pad is contacted 

* 0 1 1 . A further aspect of the invention is the structuring of associated image processing 

for a pressure-sensor array touch-pad to capture hand and foot contact postures 
and gestures 

12. A further aspect of the invention is the structuring of associated image processing 
15 for a pressure- sensor array touch-pad to derive parameters from hand and foot 

contact postures which permit the application of useful metaphors in their 
operation. 

Q 13. A further aspect of the invention is the implementation of pressure-sensor array 

kQ touch-pads, and potentially related decentralized image processing and 

networking functions, in a mini-array chip which can be tiled into arbitrary 

Ifl shapes, potentially including instrument keys. 

i y 

e p 14. A further aspect of the invention is using key displacement together with contact 
13 position to derive at least three parameters from a standard Western keyboard key. 

25 1 5. A further aspect of the invention is a foot controller with buttons and pedals that 

have associated alphanumeric displays. 

]^ 16. A further aspect of the invention is a foot controller with any one or more of: 

hierarchical organization of changeable stored program elements, arbitrary button 
assignment of hierarchy control functions, and/or multiple interpretation 
geometric layout of buttons and pedals. 



3D 



1 7. A further aspect of the invention is a method for doing one handed drum rolls with 
acoustic drums or multiple parameter electronic drumpads. 

18. A further aspect of the invention is: mallets, beaters, and bows with any one or 
more of impact, grip, position, or pressure, strain, and/or motion sensors. 

1 9. A further aspect of the invention is an autoharp adaptation with both strings and 
strumpads. 

20. A further aspect of the invention is: a string autoharp adaptation where chord 
40 buttons issue control signals. 

2 1 . A further aspect of the invention is an autoharp adaptation where a note-oriented 
keyboard is used to replay multiple note chord buttons, potentially where the keys 
are multiple parameter keys. 

45 
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22. A further aspect of the invention is: autoharp, Pipa, Koto, Harp, Mbira, pedal 
steel, and Sitar adaptations with separate pickups for each vibrating element, 
potentially also employing pitch shifting on selected vibrating element. 

23. A further aspect of the invention is: Pipa, Koto, Harp, Mbira, pedal steel, and Sitar 
adaptations with strum-pads. 

24. A further aspect of the invention is: guitar, Pipa, Koto, Harp, Mbira, pedal steel, 
and Sitar adaptations with vibrating element excitation drivers built into the 
instrument. 

25. A further aspect of the invention is: guitar, Pipa, Koto, Harp, Mbira, pedal steel, 
and Sitar adaptations with additional string arrays and/or one or more miniature 
keyboards with keys close to the string array. 

26. A further aspect of the invention is the use of vowel synthesis in conjunction with 
a bowed instrument. 

27. A further aspect of the invention is attaching a video camera to an instrument. 

28. A further aspect of the invention is the use of optical pickups for metalaphones 
and drum heads. 

29. A further aspect of the invention is the use of non-equilibrium chemical reactions 
as musical controllers or parts of instruments. 

30. A further aspect of the invention is the use of photoacoustic phenomena as 
musical controllers or parts of instruments. 

3 1 . A further aspect of the invention is the use of video cameras as musical controllers 
and/or instruments. 

32. A further aspect of the invention is a wide variety of new signal processing 
innovations, including spatial timbre construction, hysteretic waveshaping, 
layered signal processing, location modulation of signal pan constellations, cross- 
product octave chains. 

33. A further aspect of the invention is the provision for a wide variety of control 
signal monoatic and polyadic operations as listed in the disclosure. 

34. A further aspect of the invention is the provision for a wide variety of control 
routing capabilities as listed in the disclosure, including routing at MIDI message 
index levels. 

The system and method herein can be applied to live performance (music, dance, theater, 
performance works, etc.), recorded audio and video production, and composition. 

The invention will be described in greater detail below with reference to the accompanying 
drawings. 
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Figure 1 shows a general overview of the invention. 

Figure 2 shows examples of internal interconnections among the functional grouping ele- 
ments within an instance of a signal routing, processing, and synthesis entity, all shown in 
Figure 1. 

Figure 3 shows an example of a proximate keyboard array. 

Figure 4 shows an example of an instrument-mounted miniature keyboard configuration 
employing one miniature keyboard, in particular an adapted Indian sitar with many addi- 
tional example instrument elements. 

Figure 5 shows an example of an instrument-mounted miniature keyboard configuration 
employing two miniature keyboards, in particular an adapted electric guitar with many ad- 
ditional example instrument elements. 

Figure 6 illustrates an arrangement where a dedicated continuous or near-continuous sensor 
is attached to each key so as to instantaneously measure the displacement of the attached 
key. 

Figure 7 illustrates an arrangement by which programmable tactile feedback can be applied 
to a key, either in conjunction or without a continuous or near-continuous sensor to measure 
key displacement. 

Figure 8 illustrates a shared scanning arrangement supporting a plurality of any of key- 
boards, strum -pad, buttons, switches, etc. 

Figure 9 illustrates an example method for realizing a flexible generalized strum-pad ele- 
ment and associated stored program control. 

Figure 1 0 shows an example implementation of both generalized and specific control sig- 
nals derived from panel controls, actuators, and sensors using MIDI. 

Figure 1 1 shows an example of how two independent contact points can be independently 
discerned, or the dimensional-width of a single contact point can be discerned, for a resis- 
tance null/contact controller with a single conductive contact plate or wire and one or more 
resistive elements whose resistance per unit length is a fixed constant through each resistive 
element. 

Figure 12 shows an example implementation of both generalized and specific control sig- 
nals derived from electrical contact touch-pads employing MIDI messages as the output 
control signal format. 

Figure 1 3 shows how a pressure-sensor array touch-pad can be combined with image pro- 
cessing to assign parameterized interpretations to measured pressure gradients and output 
those parameters as control signals. 

Figure 14 illustrates the positioning and networking of pressure sensing and processing 
"mini-array" chips in both larger contiguous structures and in isolated use on instrument 
keys, instrument fingerboards, and instrument bodies. 

Figure 15 illustrates the pressure profiles for a number of example hand contacts with a 
pressure-sensor array. 
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Figure 16 illustrates how six degrees of freedom can be recovered from the contact of a sin- 
gle finger. 

Figure 17 illustrates the regions of vowel sounds associated with particular resonant fre- 
quency combinations in vowel sound production. 

Figure 1 8 illustrates an example two-dimensional timbre space in traditional instrument or- 
chestration. 

Figure 19 shows an example of keys from a traditional Western keyboard fitted with mul- 
tiple uniformly- sized pressure-sensing and processing "mini -array" chips. 

Figure 20 shows electromagnetic, Hall-effect, piezo, and optical pickup methods for deriv- 
ing separate audio signals for each vibrating element of a multiple vibrating element instru- 
ment entity. 

Figure 2 1 shows how an off-bridge buzz-plate can be combined with a piezo bridge sensor 
in replacement of a gradient buzz-bridge so as to permit the use of non-ferromagnetic 
strings. 

Figure 22 shows the basic idea of controlled feedback as used in recent contemporary mu- 
sic. 

Figure 23 shows an example implementation of a simple approach for replacing acoustic 
excitation of a vibrating element with electromagnetic excitation. 

Figure 24 shows various combinations of piezo and electromagnetic vibrating element 
pickups and exciters for separately controllable excitation of each vibrating element. 

Figure 25 shows adding signal processing for spectral and amplitude control of electromag- 
netic excitation. 

Figure 26 shows multiple vibrational elements with common electromagnetic excitation. 

Figure 27 illustrates examples of single, double, and quadruple touch-pad instruments with 
pads of various sizes and supplemental instrument elements. 

Figure 28 illustrates some enhanced foot-pedal arrangements which permit simultaneous 
single-foot adjustment of a plurality of continuous range parameters for use with floor con- 
trollers. 

Figure 29 shows some example layouts involving 2 geometric regions for a moderate num- 
ber of foot operated controllers and 4 geometric regions for a larger number of foot operat- 
ed controllers. 

Figure 30 shows an example large-scale arrangement of two impact sensors and/or touch 
pads for executing one-handed drum-rolls and deriving large amounts of control informa- 
tion. 

Figure 3 1 shows an example of an enhanced autoharp implementation as provided for in 
the invention. 

Figure 32 shows how the autoharp arrangement of Figure 31 can be adjusted to replace its 
chord button array and associated strum-pads with a keyboard and one or more strum-pads 
positioned over the keyboard. 
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Figure 33 shows an example Koto implementation provided for in accordance with the in- 
vention. 

Figure 34 shows an example Mbira implementation provided for in accordance with the in- 
vention. 

Figure 35 shows an example electric guitar implementation in accordance with the inven- 
tion based on a Gibson model ES-335 guitar; the invention's enhancements shown can be 
added on as modules, added collectively, or built-in. 

Figure 36 shows an example of an adapted European arch-lute with a mix of single strings 
and double string pairs. 

Figure 37 shows an example pedal steel guitar adaptation as provided for by the invention. 

Figure 38 shows an example flat-necked instrument with a five string section emulating a 
sitar string arrangement and several additional strings used for bass or other accompani- 
ment. 

Figure 39 shows an example multiple-pitch sympathetic/buzz/twang resonator using banks 
of short audio delays with high resonances tuned to each selected pitch, each followed by 
a dedicated low-speed sweeping flanger with moderate resonance, a dedicated low-speed 
sweeping flanger with moderate resonance, and a low-speed auto-panner. 

Figure 40 shows an example adapted Chinese Pipa as provided for by the invention featur- 
ing a keyboard, strum-pad, touch-pad, slider array, switch array, and impact sensors. 

Figure 41 shows another example adapted Chinese Pipa as provided for by the invention 
featuring a bass string array, a harp string array, and impact sensors. 

Figure 42 shows a bow fitted with sensors to gather information from the hand, bow hairs, 
and bow motion. 

Figure 43 shows adaptations of a flute and recorder layout with pressure sensors replacing 
key sites, air turbulence measurements, and air pressure average measurements as provided 
for in the invention. 

Figure 44 shows how an optical pickup may be created for a suspended gong; this tech- 
nique may also be used for many other types of metallophones. 

Figure 45 shows example gong arrays as part of a one-hand or two-hand percussion instru- 
ment stand. 

Figure 46 illustrates spatial arrays of electrodes which may be used for measurement, as 
well as control, in two-dimensional and three-dimensional configurations. 

Figure 47 shows an arrangement where evolving chemical patterns in the dish of Figure 46 
are illuminated with light sources and visually monitored by an overhead camera for any 
one or more of control extraction, visual display, or visual recording. 

Figure 48 illustrates example optical measurements of photoacoustic phenomena in appli- 
cable materials which may be converted to electrical signals and an example electro-acous- 
tic measurement of photo-induced acoustic phenomena in applicable materials. 
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Figure 49 shows how generalized interfaces can be built in whole or via separable parts 
which may be used selectively as needed or appropriate. 

Figure 50 shows multiple vibrational elements with multi -channel transducers applied di- 
rectly to stereo or multi-channel mix-down. 

Figure 51 shows multiple vibrational elements with multi-channel transducers and individ- 
ual signal processing prior to mixing. 

Figure 52 shows addition of a control signal extraction element to the arrangement of Fig- 
ure 5 1 . 

Figure 53 shows partial mix-downs of vibrating element signals fed to a number of signal 
processors and straight-through paths in route to subsequent mix-down. 

Figure 54 shows a switch matrix assigning signals from vibrating elements to a number of 
signal processors en route to subsequent mix-down. 

Figure 55 shows a more flexible method for providing signal processors with vibrating el- 
ement signals and other signal processor outputs via switch matrix, and additional partial 
mix-downs by replacing said switch matrix with a mixer. 

Figure 56 shows configuration control of signal processors, mixers, and switch matrix, and 
synthesizer interfaces via logic circuitry and/or microprocessing. 

Figure 57 shows a very general combined environment for multi-channel signal process- 
ing, mixing, excitation, and program control of overall configuration. 

Figure 58 shows a stereo-input, stereo output configuration of two monaural flange and/or 
chorus elements wherein the unaltered signal of each input channel is combined with a de- 
lay-modulated signal from the opposite channel. 

Figure 59 illustrates a combination of a spatialized effect, two distortion elements, and a 
stereo (N=M=2) cross-channel modulated delay. 

Figure 60 illustrates examples of inhomogeneous layered signal processing which may be 
used as shown, with selected omissions, or as an archtype for similar constructions. 

Figure 61 illustrates an example of a generalized hysterisis model construction as provided 
for by the invention. 

Figure 62 shows an example implementation of a cross-product octave chain particularly 
suited to low cost implementation with logic chips or simple DSP program loops. 

Figure 63 illustrates an example flexible control and configuration hierarchy for control 
signal and stored program handling and organization. 

Figure 64 shows an example method for the generation of control signals from fundamental 
and overtone information in a signal from a vibrating element of fixed known pitch. 

Figure 65 shows combining and/or processing fundamental and overtone information ob- 
tained from a vibrating element signal prior to parameter extraction. 

Figure 66 shows an example implementation of an adaptive method for tracking overtones 
for a variable-pitch vibrating element with known overtone series. 
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Figure 67 illustrates an example approach wherein a plurality of LFOs with features as pre- 
scribed by the invention may be implemented. 

Figure 68 illustrates traditional stage lighting elements including over-heads, far-throw, 
foot, back, floor. 

Figure 69 illustrates example instrument lighting. 
Figure 70 illustrates example rotating speaker emulation light sculptures. 
Figure 71 illustrates light pyramid arrays and light columns arrays. 
Figure 72 illustrate stage video projection arrangements. 

DETAILED DESCRIPTION 

Overview 

The invention relates to a collection of instruments (adapted, electronic, or combined), gen- 
eralized instrument electrical interfaces, control signal extraction and manipulation sys- 
tems, musical synthesis modules, layered audio signal processing, lighting control, light 
sculptures, instrument lighting effects, video control, and video display. The resulting rich 
sonic and visual environment can be used for live performance, recorded audio and video 
production, and composition. 

Figure 1 shows a general overview of the invention which, at its highest level, consists of 
one or more instances of instrument entities 100, generalized interface entities 110, and sig- 
nal routing, processing, and synthesis entities 120. It is understood that the invention pro- 
vides for the possibility of several instances of each of these entities. For example, several 
instruments 100 may be supported (adapted guitar, adapted sitar, adapted autoharp, touch- 
pad/slider/switch controller, etc.) by each 110/120 system; further, for each collection of 
plurality of instruments 100 and signal routing, processing, and synthesis entities 120, there 
may be both full- feature interface cables or simplified reduced- feature cables implement- 
ing versions of the generalized interface 110; finally, for each collection of instruments 100 
and generalized interface entities 110 there may be various versions of signal routing, pro- 
cessing, and synthesis entities 120 (small performance systems, large performance systems, 
studio recording-oriented or composition-oriented systems, etc.). 

Instrument Overview 

In more detail, each instrument entity 100 in general internally consists of one or more el- 
ements. The elements fall into two broad categories, namely those that produce audio-fre- 
quency signals and those that instead produce only control signals. Of these, it is also 
possible to derive control signals from the audio-frequency signals (reflecting pitch, ampli- 
tude, relative harmonic content, etc.). Control signals, regardless of their origin, in general 
are used to control the processing, replay, or synthesis of audio-frequency signals; howev- 
er, the control signals can also be used to control lighting, video, special effects, etc. 
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Referring to Figure 1, the instrument 100 may contain internal power sources (such as bat- 
teries, large-value capacitors, etc.) and/or power regulation elements 101. Next the control 
signal sources that may, be included within an instrument entity 100 may be of a traditional 
technology or nature, such as knobs, keys, switches, touch-pads, sliders, buttons, sensors, 
etc.; these will be termed electronic interface elements 102. In addition, it is also possible 
to generate control signals from more exotic processes such as chemical oscillators, chem- 
ical chaos, photoacoustic, environmental sensors, etc. These will be termed alternative con- 
trol signal elements 103. The audio-frequency signal sources that may be included within 
an instrument also broadly fall into two classes. One class is that of traditional vibrating el- 
ements (strings, tynes, surfaces, solid volumes, air columns, etc.) whose mechanical audio- 
frequency vibrations can be electrically sensed via electromagnetic, photo-electric, piezo, 
Hall-effect, or other types of sensors or transducers. In many cases it is possible to excite 
these mechanically vibrating elements by electronic methods (magnetic fields, piezo trans- 
ducers, etc.) or electronically controlled methods (motorized bowing, solenoid strikers, 
etc.) This first class of audio-frequency signals sources will be termed sensed/excited vi- 
brating elements 104. In addition, it is also possible to generate audio-frequency signals 
from more exotic processes such as chemical oscillators, chemical chaos, photoacoustic, 
environmental sensors, etc. These will be termed alternative control signal elements 105. 
Finally, the instrument may also contain various additional video, lighting, and special ef- 
fect elements 106. 

Generalized Interface Overview 

Again referring to Figure 1 , the invention provides for both instrument entities 100 and sig- 
nal routing, processing, and synthesis entities 120 to be fitted with compatible electrical in- 
terfaces, termed generalized instrument interfaces or (or more concisely, generalized 
interfaces) 110, which can exchange any of the following: 

• incoming electrical power (111) 

outgoing control signals from switches, controls, keyboards, sensors, etc., 
typically in the form of MIDI messages but which may also involve contact 
closure or other formats (112) 

• control signals to lights, pyrotechnics, or other special effect elements within 
and/or attached to the instruments, said signals being either in the form of MIDI 
messages, contact closure, or other formats (113) 

• outgoing audio signals from individual audio-frequency elements or groups of 
audio-frequency elements within the instruments (114) 

• incoming excitation signals directed to individual audio- frequency elements or 
groups of audio-frequency elements within the instruments (115) 

• outgoing video signals (such as NTSC, PAL, SECAM) or image signals sent from 
the instrument (116) 

• incoming video signals (such as NTSC, PAL, SECAM) or image signals sent to 
the instrument for purposes such as display or as part of a visually controlled 
instrument (117). 
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The interfaces may be realized by one or more of any of connectors, cables, fibers, radio 
links, wireless optical links, etc. 

Signal Routing, Processing, and Synthesis Overview 

Referring to Figure 1 , the invention provides for one or more signal routing, processing, 
and synthesis entities 1 20. These entities first route and process received audio-frequency, 
control, and video signals. Additionally, these entities 120 may extract control signals from 
received audio-frequency and video signals, perhaps under the direction of selected control 
signals. Finally, these entities 120 may also synthesize audio-frequency, control, and video 
signals, typically under the direction of selected control signals. 

Again referring to Figure 1 , the signal routing, processing, and synthesis entities 120 inter- 
nally may include: 

power supplies 121 for both internal and instrument powering 

• control signal routing 122 for interconnecting control signal sources with control 
signal destinations 

• control signal processing 123 for instantaneous control message transformations 
(such as inversions) and inter-operations (such as averaging, adding, 
multiplication, etc.) 

• audio signal routing 124 for interconnecting audio signal sources with audio 
signal destinations 

audio signal processing 125 for (typically real-time) transformations, typically 
under real-time control via selected control signals 

• video signal routing 126 for interconnecting video signal sources with audio 
signal destinations, typically under real-time control via selected control signals 

• video signal processing 727 for (typically real-time) video signal transformations, 
potentially under real-time control via selected control signals 

• control signal extraction 128a for the derivation of (typically real-time) control 
signals from audio or video signals, potentially under real-time control via 
selected control signals 

• control signal synthesis 128b for the internal creation of time-varying control 
signals (such as low-frequency control oscillators, envelop generators, slew 
limiters, etc.), potentially under real-time control via selected control signals 

• audio signal synthesis 129a, typically under the direction of selected control 
signals, and typically as per conventional music synthesizer hardware and 
software 

• video signal synthesis 129b, typically under the direction of selected control 
signals. 

• program storage 130 for storing configuration programs and event sequences 

In Figure 1 it is understood that the elements 121 through 130 represent functional group- 
ings and not necessarily hardware-centralized or software-centralized subsystems. 



LUDW 001/01 US 



Figure 2 shows examples of internal interconnections among the functional grouping ele- 
ments 121 through 1 30 within an instance of a signal routing, processing, and synthesis en- 
tity 120. In Figure 2, as before, it is understood that the elements 121 through 130 represent 
functional groupings and not necessarily hardware-centralized or software-centralized sub- 
systems. 

In the example interconnections, power is distributed throughout via functional fan-outs 
131; here it is understood that there many be many decentralized power supplies for the in- 
dividual subsystems comprising or implementing elements 122-130. Program store infor- 
mation is also distributed throughout via paths 132 (associated with specific subsystems of 
elements 122-129) and/or path 133 to the control signal routing element 122; typically both 
methods are used as portions of the program control may be stored within individual ele- 
ments 122-129 and portions may reside within one or more centralized program store sub- 
systems (such as MidiTemp model MP-88, Digital Music Corporation model MX-8, 
controlling PC, etc.), comprising 130. 

Remaining Document Overview 

With this overview complete, the remainder of the discussion is organized as follows. The 
next four Sections concern instruments 100. First, a number of instrument element and in- 
strument subsystems are described. Two subsequent sections then describe a large number 
of example instruments that are perfected through applicable combinations and arrange- 
ments of the aforementioned instrument elements and subsystems of elements; the first of 
these sections purely electronic controllers while the second addresses adaptations of con- 
ventional instruments with special attention paid to specific nuances and opportunities 
within those instruments. Following this, some alternative audio and control signal sources 
are then considered. 

Next the general instrument interface 110 is then considered in additional detail. A subse- 
quent section then addresses the signal processing, and synthesis entities 120. A final sec- 
tion provides a few example envisioned applications of the invention. 

Instrument Elements and Instrument Subsystems 

The invention includes a number of electronically interfaced instruments used by one or 
more performers. 

These instruments involve either pure electronic interfaces arranged to form an instrument, 
vibrating elements which typically are in arrangements adapted from existing instruments, 
exotic electrically-monitored oscillatory elements (such as chemical oscillators), electronic 
or numerical chaotic models used as sources, or combinations of these laid out in an artis- 
tically operative and ergonomic fashion. Vibrating elements within an instrument may also 
be made to vibrate via electronically controlled or induced excitation from magnetic field, 
piezo electro-mechanical, or other electronically-driven or electronically-controlled excita- 
tion. 

In general an instrument consists of one or more instrument elements which may be of one 
more differing types or classes. These instrument elements may be thought of as sub- 
systems within the instrument. For example, a 6-string guitar has six vibrating strings; each 
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string is an example of a vibrating element. A single electromagnetic or piezo pickup may 
be used to amplify the entire group of six strings. The guitar may also have separate elec- 
tromagnetic or piezo pickups for each string, as is commonly done for adding a MIDI in- 
terface to an existing electric guitar. This example guitar then simultaneously has six 
5 vibrating elements, one group-pickup subsystem, and six single-string pickup subsystems. 

The guitar may be further enhanced with MIDI-command issuing controls, such as knobs, 
switches, joysticks, touch-pads, motion/position sensors, etc.; these represent an additional 
subsystem. A reduced-size musical keyboard may be added to the guitar, representing yet 
another subsystem. 

^ Specific classes of instrument elements and/or instrument subsystems are described in the 

subsections that follow. 



2.1 Electronic Interface Instrument Elements and Subsystems 

1 5 This class of instrument elements and instrument subsystems do not create audio frequency 

phenomenon directly but are rather used to control one or more music synthesizers, audio 
mixers, and/or signal processing functions. 

.'3 2.1.1 Proximate, Miniature, and Superimposed Keyboards 

It) Standard western keyboards found on pianos, harpsichords, organs, and synthesizers are 

IZ widely used as a human interface for electronic musical instruments. Some instruments, 

; J such as organs and harpsichords, have traditionally (for centuries) included two or more 

]Z such keyboards to allow the instrument player to rapidly select among two or more timbres 

i=j or ranges. The spacing of the keyboards is almost without exception found to be far enough 

25 apart that a hand must be committed uniquely to a given keyboard for the moments that the 

O keys are played. This is due to the fact that the bulk of apparatus under the keyboards, key- 

y board frame, etc. prevented the keyboards from being mounted very close together, re-en- 

H forced by the fact that music has been composed for playing at most one keyboard with a 

^ given hand (although in virtuoso pieces a given hand may very rapidly jump among key- 

3€ boards). One aspect of the invention expands the usage of traditional keyboards by remov- 

i=z ing this limitation via various means. 

2.1.1.1 Proximate Keyboard Arrays 

One method of implementation is to mount a plurality of keyboards close enough together 
35 that one hand can, to degrees determined by mechanical details, simultaneously play notes 

on two or more traditional keyboards. There are three methods for increasing the workable 
proximity of groups of keyboards: 

reduce the vertical separation of the keyboards 

40 • overhang the ends of the white keys on a higher keyboard over the backs of the 

white and black keys of a lower keyboard 

• reduce the physical length of the keys 

Many modern electronic keyboards have very shallow mechanisms and frames. It is there- 
^ fore quite straightforward to mount two or more commonly available electronic keyboards 

employing either or both of the first two methods. With some overhang and (vertically or 
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horizontally) shallow enough mechanisms, it becomes possible to play notes on both key- 
boards simultaneously. In nominal configurations the thumb-to-pinky reach is nearly the 
same across both keyboards. Clearly some finger configurations are difficult or impossible 
across the two keyboards, but there are also limitations in conventional keyboards that are 
5 incorporated in the development of established fingering technique and respected in key- 

board music composition; similar minor technique development and compositional respect 
extensions can be developed for such proximate keyboard arrays. 

Without reducing the size of the keyboards a single hand can even make invaluable use of 
three keyboards within a confined range; simple example is to add back-up notes of the 
same pitch or differing octaves. However, two hands may use the two-keyboard playing 
techniques to make avid use of a three, four, or more proximate keyboard array. 

Figure 3 shows an example of a proximate keyboard array. In this example, three keyboards 

301, 302, 303 are arranged in an overhanging staircase arrangement. Three views are 
15 shown: a side view 300, a top view with hidden key areas suggested by dashed lines 310a, 

and the side view of 300 reoriented as an orthogonal projection 310b of the top view 310a. 

The separation distances 305a, 305b between the tops of the keys of a given keyboard and 
Q the bottoms of the keys of a keyboard overhanging it should be minimized and in the limit 

'0 are just slightly larger than the maximum travel distance of the overhanging key. The depth 

|f0 of the overhang 304a, 304b is set in the balance between the trade-off of maximizing de- 

?™ sired accessibility to the back of an overhung key and minimizing the separation distance 

'fi between the edges of the keys of two adjacent proximate keyboards. It is noted that any of 

JE the keyboards used here may be either of a standard variety or any of the more advanced 

J=j keyboards described later (miniature, superimposed, multi-parameter keys, pressure-sensor 

25 array, etc.). It is also noted that this technique may be applied to other types of keyboards 

O with applicable types of key geometry. 

J^T 2.1.1.2 Miniature Keyboards 

=D If the depth of the keyboard is reduced, the span of a given hand is increased further. This 

(J) ma Y be done by making the keys relatively shorter, forming a stubby keyboard, or by 

H shrinking the size of the entire keyboard in all dimensions. Such miniature keyboards are 

commonly found on consumer electronic keyboards and keyboard instruments made for 

children. 

Clearly a proximate keyboard array can be created from miniaturized keyboards. The range 
35 of the fingers within and across individual component keyboards may be greatly increased 

in this fashion, albeit with a perhaps somewhat compromised tradition and technique. 

An additional, and particularly valuable role for the proximate capabilities of such minia- 
ture keyboards is to mount them, as a component, on an instrument with other components 
4Q so as to form a more complex instrument where free fingers can operate two or more such 

components simultaneously. As a simple example, a guitarist using a thumb-pick or classi- 
cal guitar technique can easily use free fingers to play chords, bass lines, melodies, etc. on 
a miniature keyboard attached to a guitar. 

Figure 4 shows an example of an instrument-mounted miniature keyboard configuration 
45 employing one miniature keyboard 421, in particular an adapted Indian Sitar 400 with 

many additional example instrument elements which will be described later. 
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Figure 5 shows an example of an instrument-mounted miniature keyboard configuration 
employing two miniature keyboards, in particular an adapted electric guitar 500 with many 
additional example instrument elements which will be described later. Here note the two 
keyboards 521a, 521b are proximate enough to allow both keyboards and the guitar strings 
5 to be played simultaneously. 

Clearly these methods of miniature keyboard attachment(s) can be applied to other instru- 
ments (Sitar, Pipa, Saz, pedal steel guitar, plucked string bass, etc.) as well as being used 
to create entirely new types of instruments and controllers as will be discussed herein. 

10 2.1.1.3 Superimposed Keyboards 

It is also possible to make contact-closure keyboards with multiple contact sets that actuate 
at increasing depths of key depression. Such keyboards may or may not have tactile feed- 
back as to each level of actuation. Pratt-Read manufactured a "double-touch" keyboard for 
use in home console organs which closed one set of contacts with a noticeable restoring 

* ^ pressure at about half of the possible key-displacement which persisted through full key 

displacement where another contact set closed at the end of key travel. Also, many "veloc- 

£ _ ity sense" keyboards are realized by SPDT switches actuated with each key; here the be- 

ginning of key travel opens a pair of contacts and the end of key travel closes a second set 

]«. of contacts, but with no mid-travel tactile feedback. 

20 

1=* In either case, there are one contact closure event at partial key travel and two events at full 

=y key travel. These events can be interpreted as superimposed keyboards. One example inter- 

; £ pretation is that the first event triggers one synthesizer voice and the second even triggers 

■!j a second voice; in this manner keys struck with partial displacement sound with only one 

25 voice but those struck with full displacement sound both voices. Another example is for a 

l~ first voice to be triggered at partial displacement but turned off at full displacement. If the 

I sj first voice has a long attack, it would be drowned out by the second voice, or in short dura- 

|^ tion serve as acceptable transient ornamentation (for example, mimicing a "key click" or 

=0 "air turbulence chiff this arrangement effectively resulting in a partial key displacement 

sounding only the first voice and a full key displacement sounding only the second voice. 
H Note in either arrangement, a fluctuation of the applied key pressure can vary which voices 

continue to sound (in the first arrangement, the second voice will go on and off with the 

first voice held; in the second arrangement, the first and second voices will alternate being 

on or off in a mutually exclusive fashion). 

35 As the superposition of keyboard principal proves useful in this two-level setting, it is nat- 

ural to consider further extensions of this approach to more levels and additional interpre- 
tations. In the limit, a keyboard could have a continuous sensor (such as a potentiometer, 
magnetic or optical gradient, etc.) or near-continuous sensor (such as a binary encoded con- 
trol) attached to each key. Figure 6 illustrates an arrangement where a dedicated continuous 

40 or near-continuous sensor is attached to each key so as to instantaneously measure the dis- 

placement of the attached key. In such an arrangement external electronics would define 
quantized displacement thresholds to which various superimposed keyboard interpretations 
would be assigned. 

45 As a first bonus, it is also noted that this same continuous or near-continuous key-displace- 

ment sensor arrangement can be used in other operational modes to provide other very valu- 
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able expressive functions, for example volume or timber control or velocity contour 
tracking, as will as will be described in a later section. 

In practice, the two-level superimposed keyboard provides the player with tactile feedback 
as to what point of travel the key had passed in the form of a noticeable change in resistive 
restoring pressure. For a more generalized system as described above and illustrated in Fig- 
ure 6, there may be applications where such tactile feedback is not especially necessary, for 
example in triggering additional synthesizer voices to create an increasing gradient of rich- 
ness as the key is pressed further and further. In other circumstances, particularly if there 
are only a few levels implemented, tactile feedback may indeed be desirable, particularly 
that with discernible discrete steps matching the trigger-level quantization points in key 
travel. 



Highly flexible programmable tactile feedback can be imposed separately on each key by 
a dedicated solenoid, motor, pneumatic, fluid, or other means. Less flexible yet still some- 
15 what programmable tactile feedback could also be had by means of an electrically adjust- 

able global mechanical arrangement serving all keys in a keyboard, for example engaging 
additional sets of springs or pliable rubber pressure-resisting cones. Figure 7 illustrates an 
O arrangement by which programmable tactile feedback can be applied to a key, either in con- 

junction with or without a continuous or near-continuous sensor to measure key displace- 
2q ment. Without key position information, an electrically-controlled restoring force element 

171 with built-in levels of key pressure resistance (for example, by means of a sequence of spa- 

"~ dally distributed electromagnetic coils that can be switched on at configuration time to cre- 

j=j ate additional levels of force past specific displacement depths) could be used. With key 

j5 displacement information, a simple dedicated solenoid, motor, pneumatic, fluid, or other 

25 means can be made to have its restoring force vary over the key travel in a highly flexible 

p manner. Since key travel can be fast, the transient response of the tactile feedback system 

Ife 1 must typically have a fast rise time and be free of overshoot. If electromagnetic or electric 

if field means are used to provide key displacement resistance, care must be made to shield 

^ these elements to as to not create electromagnetic transients that could leak into nearby 

5$5 electronics or music instrument pickups. 

Finally, it is pointed out that as an additional bonus, the above arrangement is also capable 
of synthesizing different types of mechanical so-called keyboard "actions", for example the 
"feel" of various types of piano manufacture keys versus harpsichord keys, etc. Thus the 
development of a keyboard with per-key continuous or near-continuous displacement mea- 
^ surements and programmable key-displacement resistance can provide an extraordinary 

level of enhancements to conventional keyboards. This can be enhanced significantly with 
the addition of pressure sensing arrays on each key as will be described later. 

2.1.1.4 Shared scanning electronics 

40 In arrangements with multiple keyboards, superimposed keyboards, or related input devic- 

es (such as the strum-pads discussed below) the keyboard-scanning electronic hardware 
can be in many cases largely shared across pluralities of these keyboard contacts and/or re- 
lated input devices. For example, a common microprocessor could be used to generate 
common multiplexing address for a group of contacts or sensors across several keyboards 

45 and the status of individual contacts would then be serially polled or transferred in parallel. 
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Figure 8 illustrates a shared scanning arrangement supporting a plurality of any of key- 
boards, strum-pad, buttons, switches, etc. 

2.1.2 Strum-pads 

5 A few early music synthesizers replaced a conventional keyboard with a low-activation 

pressure membrane switch array laid out to resemble a keyboard. One could freely tap or 
easily drag fingers over the membrane switch array without the overhead and potential in- 
jury involved in more deeply operative conventional keyboards. Because of the lack of con- 
ventional keyboard action and technique, such keyboards rapidly lost their appeal. More 

1 0 recently, the Suzuki "Omnichord" product, designed to mimic an autoharp, provided a low- 

activation pressure membrane switch array, called a "strum-pad," laid out to mimic the 
strummed-string array of an autoharp; as a selected chord button is activated various notes 
associated with the chord are assigned to the various membrane switches so that a finger 
sweeping over the strum-pad produces an arpeggiated chord in a way suggestive of strum- 

1 5 ming a traditional autoharp. The Omnichord strum-pads are hard- wired to repeat notes mul- 

tiple times and the note assignment software permits only fixed chord selections with 
preassigned arpeggio note sequences. 

Jj5 The invention includes an important element to create or expand instruments through a gen- 

eralized adaptation of these ideas: 

=~ • a more generalized strum-pad element with the following attributes: 

«g • low activation-pressure proximate switches 

ji • linear arrangement (although others are useful) 

25 • no hard-wired note repeats 

• visual and/or small tactile markings to the player 

i~ • compact physical size 

i=j • simultaneous multiple switch activation without perceivable interaction 

l ~ • generalized note event information that can be assigned interpretation under 

program control 



m 
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• more generalized strum-pad interpretation software and hardware with the 
following stored program attributes and assignments which can be rapidly altered 
during playing: 

assignment to selected melodic notes, percussive events, lighting or special 
effect events, etc. 

arpeggio pattern select 

• note-repeats added as desired and in the manner desired 

• issuance of note, outgoing program change, and/or other control signals at 
the initial activation of each stored program (to sound a background chord, 
activate lights, etc.) with or without activity on the strum-pad 
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• selection and rapid change of specific programmable attributes and assignments 
via button or foot-switch control 

The resulting element can , for example, be attached to a guitar pick-guard and used in con- 
junction with foot-switches and/or finger-activated buttons to select stored program inter- 
pretations. Free fingers can then, while freely playing the guitar as normal, "strum" or tap 
arpeggios, trigger percussion devices, trigger lighting or special effect events, etc. 

Figure 9 illustrates an example method for realizing a flexible generalized strum-pad ele- 
ment and associated stored program control. In this example implementation, the strum-pad 
1 o switches can be electrically wired to a simple conventional MIDI keyboard interface so that 

each consecutive switch triggers a consecutive MIDI note event. The note event stream is 
then directed to a MIDI message processor which can, under program control, reassign 
each incoming note event a potentially new MIDI note number and MIDI channel, or per- 
haps a null operation to create "safety" or "dead" zones. From here individual MIDI chan- 
1 5 nels can be directed to a variety of destinations: various synthesizer voice channels, lighting 

systems, special effect systems, etc. Additional control possibilities can be further realized 
by translating note events into other types of MIDI events, as described later, or into non- 
15 MIDI control signals. 

It is also possible to add note-velocity and/or M key-pressure"/"after-touch"/"channel-pres- 
U sure" control to the strum-pad by placing a velocity sensor (such as a piezo element) and/or 

jy pressure-sensor under it and feeding the resulting signal(s) to the MIDI keyboard interface 

: p as would be done in a conventional MIDI keyboard realizing these features with such sen- 

Q sors. It is also possible to supplement, or replace altogether, each membrane switch with a 

Q pressure-sensor, thus creating a pressure-sensor array. Such an array can be used to imple- 

^ ment note-velocity and/or "key-pressure'7"after-touch"/"channel-pressure" control, but can 

J=f also be used for a great many other purposes, particularly when implemented in a two-di- 

mensional array, as described later. 



2.1.3 Panel Controls, Actuators, Sensors 

Expressive control can be enhanced considerably by attaching one or more of any of vari- 
ous additional panel controls, actuators, and sensors to any electronic instrument. 

Applicable types of panel controls include potentiometers (knob, slider, etc.), joysticks, 
panel switches, panel buttons, etc. Panel controls may be distributed in isolated spots, in 
35 small groups, or in arrays. 

Applicable actuators can include limit switches, magnetic switches, mercury switches, op- 
tical detectors, piezo or other impact detectors, etc. Actuators may be attached or associated 
with moveable parts of instruments (such as guitar vibrato " whammy" bars, harp tuning le- 
vers, autoharp string-damper bars, etc.). Additionally, actuators may be affiliated with the 
40 instrument as a whole, detecting rapid jarring of the instrument etc. Further, actuators may 

also be provided in isolated spots of the instrument, such as velocity-sensitive tap-actuators 
for percussion event-triggers and "body blows" to the instrument, as abstracted from for ex- 
amples: ancient Chinese Pipa, centuries old Flamenco guitar, and recent Jimi Hen- 
drix/ Adrian Belue (borderline to actual guitar abuse) techniques. 

45 
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Applicable sensors can include pressure, motion (velocity, acceleration, etc.), position (op- 
tical, magnetic or electric field, electromagnetic standing wave, acoustic standing wave, 
etc.), impact (such as piezo sensors used with electronic drum pads), tension, strain, torsion, 
light, temperature, etc. Position sensors may be used to measure the position of a physical 
5 element of an instrument (such as a damper bar or pitch-modulating lever) or the absolute 

position of the instrument itself. Tension sensors may be used, for example, to measure 
modulated string tension as on a Koto or electric guitar; such string tension controllers need 
not even involve sounding strings — for example a small Koto string and bridge arrange- 
ment may be used strictly as an electronic control provided to the player in the form of a 
1 0 familiar Koto string format. 

In general these panel controls, actuators, and sensors can be configured to provide a range 
of either continuous or discrete-step control voltages. In some cases additional electronics 
or subsequent software transformations may be necessary to re-contour/redistribute the 
control voltage over the full range of the controls, actuators, and/or sensors. In some cases, 
multiple transformations may be made available under selectable or stored program con- 
trol. In any case, the resulting control voltages may be then treated as generalized control 
|=? signals which are presented to the generalized interface 110. Alternatively, some of the 

.'J control voltages may be used for specialized control signals, such as setting values for note- 

g| velocity, after- touch, etc. 

\f t Figure 10 shows an example implementation of both generalized and specific control sig- 

*J nals derived from panel controls 1001.1 - lOOl.n, actuators 1002.1 - 1002.ni, and sensors 

IS 1003.1 - 1003.k as provided for by the invention. The panel controls 1001.1 - lOOl.n, ac- 

tuators 1002.1 - 1002.m, and sensors 1003.1 - 1003,k may or may not be provided with ap- 
25 propriate interface electronics, respectively 1011.1 - lOILn, 1012.1 - 1012.m, and 1013.1 

p - 1013.k which deliver signals to a control signal formatter 1050 which issues control sig- 

IaJ nals 105 1 . These control signals may be of various formats, for example MIDI. 

■ p 2.1.4 Null/Contact Touch-pads 

!3;b Distinguished from panel controls and sensors considered above are what will be termed 

; ~ null/contact touch pads. This is a class of contact-position sensing devices that normally are 

in a null state unless touched and produce a control signal when touched whose signal value 
corresponds to typically one unique position on the touch-pad. Internal position sensing 
mechanisms may be resistive, capacitive, optical, standing wave, etc. Examples of these de- 

35 vices include one-dimensional-sensing ribbon controllers found on early Music synthesiz- 

ers, two-dimensional -sensing pad such as the early Kawala pad and more modern mini- 
pads found on some lap-top computers, and two-dimensional-sensing see-through touch- 
screens often employed in public computer kiosks. As a music controller these devices are 
attractive in that they can very easily capture very expressive finger nuances as does a vio- 

40 lin fingerboard or Koto bridge/string arrangement but not limit them to controlling only 

pitch. Two-dimensional versions of these devices also permit the use of spatial metaphors 
and notions of "musical finger-painting." 

The null condition, when the pad is untouched, requires and/or provides the opportunity for 
^ special handling. Some example ways to handle the untouched condition include: 

sample-hold (hold values issued last time sensor was touched, as does a joystick) 
— /0 — 



LVDW 001/01 US 

• bias (issue maximal -range value, minimal-range value, mid-range value, or other 
value) 

• touch-detect on another channel (i.e., a separate out-of-band "gate" channel) 

5 Example uses for these devices as controller elements within the context of the invention 

include any one or more of the following: 

issuance of melodic or percussion note events 

• pitch, amplitude, timbre, and location (i.e., panning, etc.) modulations 
10 • lighting and/or special effects control 

general MIDI CC control signals 

Additional enhancements can be added to the adaptation of null/contact touch pad control- 
lers as instrument elements. A first enhancement is, as discussed above for strum-pad ele- 
1 5 ments, the addition of velocity and/or pressure sensing. This can be done via global impact 

and/or pressure-sensors in the same manner as described for the strum-pads. An extreme of 
this is implementation of the null/contact touch pad controller as a pressure-sensor array; 

0 this special case and its many possibilities are described later. On the simpler extreme, a 
null/contact touch pad together with such a global velocity and/or pressure-sensor can act 

BJ as a rich metaphor for a drum head, gong surface, cymbal surface, etc. and as such may be 

L~ played with fingers, whole hands, cushioned beaters, or sticks. 

1 ii 

: p A second enhancement is the ability to either discern each dimensional-width of a single 

D contact area or, alternatively, independently discern two independent contact points in cer- 

tain types of null/contact controllers. Figure 1 1 shows an example of how two independent 
contact points can be independently discerned, or the dimensional -width of a single contact 
point can be discerned, for a resistance null/contact controller with a single conductive con- 
tact plate (as with the Kawala pad product) or wire (as in a some types of ribbon controller 
products) and one or more resistive elements whose resistance per unit length is a fixed 
constant through each resistive element. It is understood that a one-dimensional null/con- 
tact touch pad typically has one such resistive element while a two-dimensional null/con- 
tact touch pad typically has two such resistive elements that operate independently in each 
direction. Referring to Figure 1 1, a constant current source can be applied to the resistive 
element as a whole, developing a fixed voltage across the entire resistive element. When 
any portion of the resistive element is contacted by either a non-trivial contiguous width 
and/or multiple points of contact, part of the resistive element is shorted out, thus reducing 
the overall wind-to-end resistance of the resistance element. Because of the constant cur- 
rent source, the voltage developed across the entire resistive element drops by an amount 
equal to the portion of the resistance that is shorted out. The value of the voltage drop then 
equals a value in proportion to the distance separating the extremes of the wide and/or mul- 
tiple contact points. By subtracting the actual voltage across the entire resistive element 
from the value this voltage is normally, a control voltage proportional to distance separating 
the extremes of the wide and/or multiple contact points is generated. Simultaneously, the 
voltage difference between that of the contact plate/wire and that of the end of the resistive 
element closest to an extremal contact point is still proportional to the distance from said 
end to said extremal contact point. Using at most simple op-amp summing and/or differen- 
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tial amplifiers, a number of potential control voltages can be derived; for example one or 
more of these six continuously- valued signals: 

• value of distance difference between extremal contact points (or "width"; as 
described above via constant current source, nominal reference voltage, and 
differential amplifier) 

♦ center of a non-trivial- width region (obtained by simple averaging, i.e., sum with 
gain of 1/2) 

* value of distance difference between one end of the resistive element and the 
closest extremal contact point (simple differential amplifier) 

• value of distance difference between the other end of the resistive element and the 
either extremal contact point (sum above voltage with "width" voltage with 
appropriate sign) 

Further, through use of simple threshold comparators, specific thresholds of shorted resis- 
tive element can be deemed to be, for example, any of a single point contact, a recognized 
contact region width, two points of contact, etc., producing corresponding discrete- valued 
control signals. The detection of a width can be treated as a contact event for a second pa- 
rameter analogous to the single contact detection event described at the beginning. Some 
example usage of these various continuous and discrete signals are: 

♦ existence of widths or multiple contact points may be used to trigger events or 
timbre changes 

* degree of widths may be used to control degrees of modulation or timbre changes 

• independent measurement of each extremal contact point from the same end of 
the resistive element can be used to independently control two parameters. In the 
simplest form, one parameter is always larger than another; in more complex 
implementations, the trajectories of each contact point can be tracked (using a 
differentiator and controlled parameter assignment switch); as long as they never 
simultaneously touch, either parameter can vary be larger or smaller than the 
other. 

It is understood that analogous approaches may be applied to other null/contact touch pad 
technologies such as capacitive or optical. 

A third possible enhancement is that of employing a touch-screen instance of null/contact 
touch pad and position it over a video display. In this case the video display signal may be 
created either within an instrument entity 100, within the signal routing, processing, and 
synthesis entity 120, or from external sources such as stage cameras, attached computers, 
etc. The video display could for example provide dynamically assigned labels, abstract spa- 
tial cues, spatial gradients, line-of-site cues for fixed or motor-controlled lighting, etc. 
which would be valuable for use in conjunction with the adapted null/contact touch pad 
controller. 

These various methods of adapted null/contact touch pad elements can be used stand-alone 
or arranged in arrays (as in a percussion controller). In addition, they can be used as a com- 
ponent or addendum to instruments featuring other types of instrument elements. Figure 12 
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shows an example implementation of both generalized and specific control signals derived 
from electrical contact touch-pads employing MIDI messages as the output control signal 
format. 

5 2.1.5 Pr ssur -S nsor Array Touch-pads 

The invention provides for the selective inclusion of considerably advanced expressive 
control of electronic musical processes through use of a pressure-sensor array arranged as 
a touch-pad together with associated image processing. As with the null/contact controller, 
these pressure-sensor array touch-pads may be used stand-alone, organized into an array of 
10 such pads, and/or used as a component and/or addendum to instruments employing other 

types of instrument elements. 

It is noted that the inventor's original vision of the below described pressure-sensor array 
touch-pad was for applications not only in music but also for computer data entry, computer 
simulation environments, and real-time machine control, applications to which the below 
described pressure-sensor array touch-pad clearly can also apply. 

A pressure-sensor array touch-pad of appropriate sensitivity range, appropriate "pixel" res- 

□ olution, and appropriate physical size is capable of measuring pressure gradients of many 
*-Q parts of the flexibly-rich human hand or foot simultaneously. Figure 1 3 shows how a pres- 

sure- sensor array touch-pad can be combined with image processing to assign parameter- 
:f t ized interpretations to measured pressure gradients and output those parameters as control 

E Jf signals. 

□ The pressure-sensor "pixels" 1300 of a pressure-sensor array touch-pad 1301 are interfaced 
P to a data acquisition stage 1302. The interfacing method may be fully parallel but in prac- 
^ tice may be advantageously scanned at a sufficiently high rate to give good dynamic re- 
j=f sponse to rapidly changing human touch gestures. To avoid the need for a buffer amplifier 
!*f for each pressure-sensor pixel 1300, electrical design may carefully balance parasitic ca- 
: ~ pacitance of the scanned array with the electrical characteristics of the sensors and the scan 

rates; electrical scanning frequencies can be reduced by partitioning the entire array into 
distinct parts that are scanned in parallel so as to increase the tolerance for address settling 
times and other limiting processes. Alternatively, the pressure-sensor array 1301 may be 
fabricated in such a way that buffer amplifier arrays can be inexpensively attached to the 
sensor array 1301, or the sensors 1300 may be such that each contains its own buffer am- 
plifier; under these conditions, design restrictions on scanning can be relaxed and operate 
at higher speeds. Although the pressure-sensors may be likely analog in nature, a further 
enhancement would be to use digital-output pressure-sensor elements or sub-arrays. A par- 
ticularly useful example of sensor sub-arrays is presented in a few paragraphs. 

The data acquisition stage 1302 looks for sensor pixel pressure measurement values that 
40 exceed a low-level noise-rejection/deformity-rejection threshold. The sufficiently high 

pressure value of each such sensor pixel 1300 is noted along with the relative physical lo- 
cation of that pixel (known via the pixel address). This noted information may be stored 
"raw" for later processing and/or may be subjected to simple boundary tests and then folded 
into appropriate running calculations as will be described below. In general, the pressure 
45 values and addresses of sufficiently high pressure value pixels are presented to a sequence 

of processing functions which may be performed on the noted information: 
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contiguous regions of sufficiently high pressure values are defined (a number of 
simple run-time adjacency tests can be used; many are known - see for example 
[Ronse; Viberg; Shaperio; Hara]) 

the full collection of region boundaries are subjected to classification tests; in 
cases a given contiguous region may be split into a plurality of tangent or co- 
bordered independently recognized regions 

• various parameters are derived from each independent region, for example 
geometric center, center of pressure, average pressure, total size, angle-of- 
rotation-from-reference for non-round regions, second-order and higher-order 
geometric moments, second-order and higher-order pressure moments, etc. 

• assignment of these parameters to the role of specific control signals (note events, 
control parameters, etc.) which are then output to the signal routing, processing, 
and synthesis entity 120; for example, this may be done in the form of MIDI 
messages. 

Because of the number processes involved in such a pipeline, it is advantageous to follow 
a data acquisition stage 1302 with one or more additional processing stages 1303. Of the 
four example processing functions listed above, the first three fall in the character of image 
processing. It is also possible to do a considerable amount of the image processing steps 
actually within the data acquisition step, namely any of simple adjacency tests and folding 
selected address and pressure measurement information into running sums or other running 
pre-calculations later used to derive aforementioned parameters. The latter method can be 
greatly advantageous as it can significantly collapses the amount of data to be stored. 

Regardless of whether portions of the image processing are done within or beyond the data 
acquisition stage, there are various hardware implementations possible. One hardware ap- 
proach would involve very simple front-end scanned data acquisition hardware and a single 
high- throughput microprocessor/signal -processor chip. Alternatively, an expanded data ac- 
quisition stage may be implemented in high-performance dedicated function hardware and 
this would be connected to a lower performance processor chip. A third, particularly advan- 
tageous implementation would be to implement a small pressure-sensor array together with 
data equitation and a small processor into a single low-profile chip package that can be laid 
as tiles in a nearly seamless larger array. Such "mini-array" chips have additional value as 
they can readily be put on instrument keys (as described below), instrument fingerboards, 
instrument bodies, etc. In such an implementation all image processing could in fact be 
done via straightforward partitions into message-passing distributed algorithms. One or 
more individual chips could direct output parameter streams to an output processor which 
would organize and/or assign parameters to output control channels, perhaps in MIDI for- 
mat, perhaps in a programmable manner under selectable stored program control. A tiled 
macro array of such "sensor mini-array" chips could be networks by a tapped passive bus, 
one- or two-dimensional mode active bus daisy-chain, a potentially expandable star-wired 
centralized message passing chip or subsystem, or other means. Creating a large surface 
from such "tile chips" will aid in the serviceability of the surface. Since these chips can be 
used as tiles to build a variety of shapes, it is therefore possible to leverage a significant 
manufacturing economy-of-scale so as to minimize cost and justify more extensive feature 
development. Advanced seating and connector technologies, as used in lap-tops and other 
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high-performance miniature consumer electronics, can be used to minimize the separation 
between adjacent chip "tiles" and resultant irregularities in the tiled-surface smoothness. A 
tiled implementation may also include a thin rugged flexible protective film that separates 
the sensor chips from the outside world. Figure 14 illustrates the positioning and network- 
ing of pressure sensing and processing "mini-array" chips in both larger contiguous struc- 
tures and in isolated use on instrument keys, instrument fingerboards, and instrument 
bodies. 

With the perfection of a translucent pressure-sensor array, it further becomes possible for 
translucent pressure-sensor arrays to be laid atop aligned visual displays such as LCDs, flo- 
rescent, plasma, CRTs, etc. as was discussed above for null/contact touch-pads. The dis- 
plays can be used to label areas of the sensor array, illustrate gradients, etc. Note that in the 
"tile chip" implementation, monochrome or color display areas may indeed be built into 
each chip. 

Returning now to the concept of a pressure-sensor array touch-pad large enough for hand- 
operation: examples of hand contact that may be recognized, example methods for how 
these may be translated into control parameters, and examples of how these all may be used 
are now described. In the below the hand is used throughout as an example, but it is under- 
stood that the foot or even other body regions, animal regions, objects, or physical phenom- 
ena can replace the role of the hand in these illustrative examples. 

Figure 15 illustrates the pressure profiles for a number of example hand contacts with a 
pressure-sensor array. In the case 1500 of a finger's end, pressure on the touch pad pres- 
sure-sensor array can be limited to the finger tip, resulting in a spatial pressure distribution 
profile 1501; this shape does not change much as a function of pressure. Alternatively, the 
finger can contact the pad with its flat region, resulting in light pressure profiles 1502 which 
are smaller in size than heavier pressure profiles 1503. In the case 1504 where the entire 
finger touches the pad, a three-segment pattern (1504a, 1504b, 1504c) will result under 
many conditions; under light pressure a two segment pattern (1504b or 1504c missing) 
could result. In all but the lightest pressures the thumb makes a somewhat discernible shape 
1505 as do the wrist 1506, cuff 1507, and palm 1508; at light pressures these patterns thin 
and can also break into disconnected regions. Whole hand patterns such the fist 7577 and 
flat hand 7572 have more complex shapes. In the case of the fist 7577, a degree of curl can 
be discerned from the relative geometry and separation of sub-regions (here depicted, as an 
example, as 1511a, 1511b, and 1511c). In the case of the whole flat hand 7500, there can 
be two or more sub-regions which may be in fact joined (as within 1512a) and/or discon- 
nected (as an example, as 7572a and 1512b are); the whole hand also affords individual 
measurement of separation "angles" among the digits and thumb (1513a, 1513b, 1513c, 
1513d) which can easily be varied by the user. 

Relatively simple pattern recognition software can be used to discern these and other hand 
contact patterns which will be termed "postures." The pattern recognition working together 
with simple image processing may, further, derive a very large number of independent con- 
trol parameters which are easily manipulated by the operating user. In many cases it may 
be advantageous to train a system to the particulars of a specific person's hand(s) and/or 
specific postures. In other situations the system may be designed to be fully adaptive and 
adjust the a persons hand automatically. In practice, for the widest range of control and ac- 
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curacy, both training and ongoing adaptation may be useful. Further, the recognized pos- 
tures described thus far may be combined in sequence with specific dynamic variations 
among them (such as a finger flick, double-tap, etc.) and as such may be also recognized 
and thus treated as an additional type of recognized pattern; such sequential dynamics - 
among postures will be termed "gestures." The admission of gestures further allows for the 
derivation of additional patterns such as the degree or rate of variation within one or more 
of the gesture dynamics. Finally, the recognized existence and/or derived parameters from 
postures and gestures may be assigned to specific outgoing control signal formats and rang- 
es. Any training information and/or control signal assignment information may be stored 
and recalled for one or more players via stored program control. 

For each recognized pattern, the amount of information that can be derived as parameters 
is in general very high. For the human hand or foot, there are, typically, artifacts such shape 
variation due to elastic tissue deformation that permit recovery of up to all six degrees of 
freedom allowed in ,an object's orientation in 3-space. Figure 16 illustrates how six degrees 
of freedom can be recovered from the contact of a single finger. In the drawing, the finger 
1600 makes contact with the touch-pad 1601 with its end segment at a point on the touch- 
pad surface determined by coordinates 1611 and 1612 (these would be, for example, 
left/right for 1611 and forward/backward for 1612). Fixing this point of contact, the finger 
1600 is also capable of rotational twisting along its length 1613 as wall as rocking back and 
forth 1614. The entire finger can also be pivoted with motion 1615 about the contact point 
defined by coordinates 1611 and 1612. These are all clearly independently controlled ac- 
tions, and yet it is still possible in any configuration of these thus far five degrees of free- 
dom, to vary the overall pressure 1616 applied to the contact point. Simple practice, if it is 
even needed, allows the latter overall pressure 1616 to be independently fixed or varied by 
the human operator as other parameters are adjusted. 

In general other and more complex hand contacts, such as use of two fingers, the whole 
hand, etc. forfeit some of these example degrees of freedom but often introduce others. For 
example, in the quite constrained case of a whole hand posture, the fingers and thumb can 
exert pressure independently (5 parameters), the finger and thumb separation angles can be 
varied (4 parameters), the finger ends 1504a can exert pressure independently from the 
middle 1504b and inner 1504c segments (4 parameters), the palm can independently vary 
its applied pressure (1 parameter) while independently tilting/rocking in two directions (3 
parameters) and the thumb can curl (1 parameter), yielding 17 instantaneously and simul- 
taneously measurable parameters which are independently adjustable per hand. Complex 
contact postures may also be viewed as, or decomposed into, component sub-postures (for 
example here, as flat-finger contact, palm contact, and thumb contact) which would them 
derive parameters from each posture independently. For such complex contact postures, 
recognition as a larger compound posture which may then be decomposed allows for the 
opportunity to decouple and/or renormalize the parameter extraction in recognition of the 
special affairs associated with and constraints imposed by specific complex contact pos- 
tures. 

It is noted that the derived parameters may be pre-processed for specific uses. One example 
of this would be the quantization of a parameter into two or more discrete steps; these could 
for example be sequentially interpreted as sequential notes of a scale or melody. Another 
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example would be that of warping a parameter range as measured to one with a more mu- 
sically expressive layout. 

Next examples of the rich metaphorical aspects of interacting with the pressure-sensor ar- 
ray touch-pad are illustrated. In many cases there may be one or more natural geometric 
metaphor(s) applicable, such as associating left-right position, left-right twisting, or left- 
right rotation with stereo paning, or in associating overall pressure with volume or spectral 
complexity. In more abstract cases, there may be pairs of parameters that go together — 
here, for example with a finger end, it may be natural to associate one parameter pair with 
(left/right and forward/backward) contact position and another parameter pair with 
(left/right and forward/backward) twisting/rocking. In this latter example there is available 
potential added structure in the metaphor by viewing the twisting/rocking plane as being 
superimposed over the position plane. The superposition aspect of the metaphor can be 
viewed as an index, or as an input-plane/output-plane distinction for a two-input/two-out- 
put transformation, or as two separated processes which may be caused to converge or mor- 
ph according to additional overall pressure, or in conjunction with a dihedral angle of 
intersection between two independent processes, etc. 

Next, examples of the rich syntactical aspects of interacting with the pressure-sensor array 
touch-pad are illustrated. Some instruments have particular hand postures naturally associ- 
ated with their playing, particularly hand drums and especially Persian and Indian hand 
drums (such as the tabla/baya bols, dumbek, etc.). It is natural then to recognize these clas- 
sical hand-contact postures and derive control parameters that match and/or transcend how 
a classical player would use these hand positions to evoke and control sound from the in- 
strument. Further, some postures could be recognized either in isolation or in gestural-con- 
text as being ones associated with (or assigned to) percussion effects while remaining 
postures may be associated with accompanying melodies or sound textures. As an addition- 
al syntactic aspect, specific hand postures and/or gestures may mapped to specific selected 
assignments of control signals in ways affiliated with specific purposes. For example, fin- 
ger ends may be used for one collection of sound synthesis parameters, thumb for a second 
potentially partially overlapping collection of sound synthesis parameters, flat fingers for a 
third partially-overlapping collection, wrist for a fourth, and cusp for a fifth, and fist for a 
sixth. In this case it may be natural to move the hand through certain connected sequences 
of motions; for example: little finger end, still in contact, dropping to flat-finger contact, 
then dropping to either palm directly or first to cusp and then to palm, then moving to wrist, 
all never breaking contact with the touch-pad. Such permissible sequences of postures that 
can be executed sequentially without breaking contact with the touch-pad will be termed 
"continuous grammars." Under these circumstances it is useful to set up parameter assign- 
ments, and potentially associated context-sensitive parameter renormalizations, that work 
in the context of selected (or all available) continuous grammars. For example, as the hand 
contact evolves as being recognized as one posture and then another, parameters may be 
smoothly handed-over in interpretation from one posture to another without abrupt chang- 
es, while abandoned parameters either hold their last value to return to a default value (in- 
stantly or via a controlled envelope). 

Now a number of example applications of the pressure-sensor array touch-pad are provid- 
ed. A natural start for a first example is that of the Indian tabla and baya; here the traditional 
bols are recognized and used to control synthesized or sample-playback sound generation. 
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The produced sound can be authentic or transcend the classical instrument. Additional pos- 
ture and gesture recognition can be added in either sound generation style to expand the 
available sounds and/or control additional signal processing such as location modulation, 
muffling or peaking filtering, reverb, sustain, instrument pitch, etc. Considering hand 
drums more generally it is noted that whole-hand slaps are commonly used in the technique 
but that the spread of the fingers in the hand slap or hand after-touch of the drum head typ- 
ically provide no usable control. With the system described above, details of at least four 
parameters of finger spread and even more on whole-hand posture in whole-hand slaps and 
ongoing after-touch pressing may be used for extensive timbre variation. 

Next, examples are given as to how derived parameters may be used to control musical pro- 
cesses and lighting control, effectively allowing one to "fingerpaint" with sound and/or 
light. There are a large number of ways in which six parameters of synthesizer "voices" 
may be controlled with one finger. One possible example of a mapping is to use all six pa- 
rameters to control prominent features of a single synthesizer voice: 

• left/right position: pitch 

• in/out position: volume 

• left/right twist: waveform morphing dimension 1 ("duty cycle," even-harmonic 
content, etc.) 

• in/out rock: waveform morphing dimension 2 ("waveform curvature," odd- 
harmonic content, etc.) 

• rotation: stereo pan 

overall pressure: filter opening 
Another example is that of controlling two voices with one finger: 

• left/right position: pitch of voice 1 

• in/out position: pitch of voice 2 

• left/right twist: pan or filter opening of voice 1 

• in/out rock: pan or filter opening of voice 2 

• rotation: relative volume balance of voice pair 

• overall pressure: total volume of voice pair 

By assigning pitch to an aspect of physical contact that is geometrically large (i.e, position 
on the pad), it is possible to get a great deal of accuracy in pitch control. In potentially typ- 
ically cases where pitch choices are to be associated with traditional scales, the pitch con- 
trol parameter may be quantized into discrete steps and each step assigned to a note in a 
scale or melody. At the point of contact within a selected quantization interval, a small "vi- 
brato" neighborhood may then be defined so that wiggling the finger position is mapped to 
a vibrato-range pitch variation (as on a violin string). If the spatially-quantized positions 
are mapped to notes in a melody, it is possible to set up mappings for several musical phras- 
es or in fact an entire melodic line start-to-finish. In the latter circumstance, it may be de- 
sirable to either "page" the pitch assignments to give up one of the position parameters for 
sound control or instead use it for laying out the melody geometrically as per a sheet of mu- 
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sic; here the spatial quantization may be uniformly spaced or under limited degrees be pro- 
portional to the pitch duration of the associated note. The sheet-music layout is particularly 
interesting because it allows the performer to concentrate extreme dexterity in the timbre 
and timing expression of a melody without having to devote very much effort or attention 
5 to the selection of pitch value. The resulting allocation shift of performer attention is very 

valuable as the amount of expression and variations in timbre are often what distinguish a 
spellbinding performance from a run-of-the-mill performance. Although purist musicians 
may scoff at the release from pitch selection struggles endemic in musical instruments over 
the centuries, they are also known to spend thousand of dollars on finest-instruments that 

10 allow additional nuances of expression and spend many, many years of their lives making 

pitch selection efforts nearly as subconscious as this instrument approach does. This class 
of instrument controller, then, allows those years of skill development to be devoted direct- 
ly to perfecting advanced degrees of musical expression, potentially higher than may be 
achieved with conventional human life spans, traditional real-time instruments, and orches- 

15 tra-conductor protocol. 

Leaving higher callings in music for the moment, it is also possible to use the pressure-sen- 
^ sor array touch-pad for lighting control, particularly multi-channel lighting and/or motor- 

'■•i controlled (any one or more of pan, tilt, zoom, gel, pattern-gel orientation, etc.) lighting. In 

multiple-light control situations, regions of the pad may be quantized into cells, each asso- 
ciated with a particular light and parameters within the region, controlling any of: light, 
brightness, position, zoom, gel, gel-pattern-orientation, etc. What can be especially inter- 
esting in performance is to combine music process control with lighting control. Some pos- 
tures, gestures, or pad-regions may be exclusively devoted to only music control or only 
lighting control parameters, but other postures, gestures, or pad-regions may be set up to 
intermingle and/share parameter assignments between music and lights. 

jj It is also known to be possible and valuable to use the aforementioned pressure-sensor array 

touch-pad, implicitly containing its associated data acquisition, processing, and assignment 

=0 elements, for many, many non-musical applications such as general machine control and 

computer workstation control. One example of machine control is in robotics: here a finger 

!= might be used to control a hazardous material robot hand as follows: 

left/right position: left/right hand position 
in/out position: in/out hand position 
in/out rock: up/down hand position 
rotation: hand grip approach angle 
overall pressure: grip strength 

left/right twist: gesture to lock or release current grip from pressure control 

A computer workstation example may involve a graphical Computer- Aided Design appli- 
cation currently requiring intensive mouse manipulation of parameters one or two at a time: 

• left/right position: left/right position of a selected symbol in a 2-D CAD drawing 
45 • in/out position: up/down position of a selected symbol in 2-D CAD drawing 

• left/right twist: symbol selection — left/right motion through 2-D pallet 
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• in/out rock: symbol selection — up/down motion through 2-D pallet 
rotation: rotation of selected symbol in the drawing 

• overall pressure: sizing by steps 

• tap of additional finger: lock selection into drawing or unlock for changes 

• tap of thumb: undo 

• palm: toggle between add new object and select existing object 

Clearly a symbol can be richly interactively selected and installed or edited in moments as 
opposed to tens to hundreds of seconds as is required by mouse manipulation of parameters 
one or two at a time and the necessary mode-changes needed to change the mouse action 
interpretation. 

Multi-Parameter Instrument Keys 

The famous multiple tape-loop Melletron product had keys which served to a rough extent 
as per-note volume controls, allowing valuable relative voice level variations. Robert Moog 
patented a key with a two-dimensional touch sensor on a keyboard key surface. The present 
invention allows for the synergistic combination of these technologies so as to create a 
three-parameter controlling key particularly suited to vowel-choir synthesis and other ap- 
plications, next extends this to include more arbitrary instrument keys (such as those on a 
woodwind), and finally develops multi -parameter sensing keys further by incorporation of 
the aforementioned pressure-sensor array touch-pad on each key. 

When voice choirs are used as instrumentation rather than the deliverer of libretto, the prin- 
cipal parameters are typically the vowel sound used and the relative amplitude of each vo- 
cal line. If these parameters were to be controlled by a keyboard, and for the moment if 
unisons of two or more vocal lines were excluded (unisons will in fact be handled later), 
each vocal line would be at a different pitch from the others. This allows at any particular 
instant specific keys on a keyboard to be uniquely associated with one vocal line apiece. As 
with the now somewhat traditional Melletron, the displacement of key sounding the note of 
a particular vocal line then may be used to control the volume of that vocal line. By incor- 
porating a two-dimensional touch-pad controller to each key, it is also possible to select and 
in fact vary the vowel sound. In phonetics and vocal pedagogy it is well known [Appelman, 
Winckel] that the quality of the vowel is largely determined by the frequencies of resonanc- 
es produce by the vocal cavity. In fact, the full range of realistic vowel sounds may be cre- 
ated by passing simple sawtooth or narrow-wide pulse oscillator waveforms into a pair of 
band emphasis filters, the vowel sounds varying as the filter emphasis frequencies are var- 
ied. Figure 17, adapted from Winckel, illustrates the regions of vowel sounds associated 
with particular resonant frequency combinations in vowel sound production. Clearly there 
are two dimensions, then, which control vowel quality at this level of approximation, and 
further the surface of the key may be viewed as a metaphor for the plot of Figure 1 7. Further 
details of effective choir synthesis and variations upon it are discussed later, but the syner- 
gistic value of the two-dimensional touch-pad key surface and key displacement as sources 
of control signals for choir synthesis is clearly established. In fact, this three-parameter per 
individual synthesizer "voice" may be very valuable in at least two additional situations. 



LVDW 001/01 US 



In a first of these additional situations, it is first noted that in traditional multiple-instrument 
orchestration, the principal parameters are volume and timbre. Using the aforementioned 
three-parameter key arrangement, key displacement may again be used for per-note volume 
control, leaving the remaining two dimensions for timbre control. Wessel and others have 
shown empirically that continuous multidimensional "timbre spaces" are useful organiza- 
tions for analyzing and executing orchestration aspects of timbre assignment. Often two- 
dimensional timbre spaces offer a more than rich enough environment to be very useful. 
Figure 18, adapted from Wessel, illustrates some example two-dimensional timbre spaces 
from traditional instrument orchestration. Again, a metaphor may be made between these 
two-dimensional graphs and the two-dimensional touch-pad key surface. The implementa- 
tion on the synthesis side may be implemented by methods as simple as volume cross-fad- 
ing of sampled traditional instruments (and/or synthesized sounds) to methods as 
sophisticated as morphable numeric instrument models. 

The second additional situation also pertains to so-called model-based synthesis (as em- 
ployed in the Yahama VL1) but over a lesser range of timbre variation, in fact a range typ- 
ically within the scope associated with a single instrument rather than a multi-instrument 
orchestration environment. Model-based synthesis typically has an abundance of parame- 
ters and a dearth of effective methods for controlling them. Selected parameters, in fact, are 
controlled with global control interfaces such as a wheel, joystick, or breath controller. Be- 
cause of the need for associating parameter control with each note, rather than a group of 
notes, the best model-based synthesis engines then have been monophonic (i.e., only pro- 
ducing one note at a time). The invention's provision of a keyboard with the availability of 
three parameters tied specifically and independently to each key is an ideal solution to a 
polyphonic model-based synthesis instrument. 

It is noted that choir synthesis, dynamic timbre-space based orchestration, and polyphonic 
model-based synthesis instruments require the synergistic combination of key displace- 
ment and key surface touch-pad, while the said combination also is fully capable of imple- 
menting Moog's original vision for two-dimensional synthesis control (filter parameters, 
oscillator waveforms, etc.) and as an alternative implementation to MIDI keyboard channel 
pressure which typically requires each active key to be fully displaced. 

The invention also provides for the application control discussed above to be enhanced yet 
further by placing a pressure-sensor array touch-pad on each key. In the limit, this would 
allow each key to derive up to six parameters for each point of contact on a key and even 
multiple points of contact (i.e., more than one finger) per key. Although custom pressure- 
sensor array touch-pads could be crafted for the keys, it is advantageous to employ the 
aforementioned pressure sensing and processing "mini-array" chips. In fact, applications to 
key surfaces could be used to dictate the canonical dimensions of the chips, for example the 
width of the top surface of a black key and a length that is a least common multiple of a 
black key surface length and a white key surface length. 

Figure 19 shows an example of keys from a traditional Western keyboard fitted with mul- 
tiple uniformly-sized pressure-sensing and processing "mini-array" chips. These chips may 
be interconnected using the networking features described earlier. Alternatively, special 
pressure-sensing and processing "mini-array" keys may be made without the chip as a tiling 
sub-component; these would be networked in the same fashion. As to how the uniformly- 
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sized pressure-sensing and processing M mini-array" chips could be applied to the keys, Fig- 
ure 19 first shows the collection of key shapes 1900 used to make a conventional Western 
keyboard. There are five types of white key shapes (1901-1905) and one black key shape 
1900 used, although the black key 1900 may have a taper leading from its widest base area 
5 1906a to a narrower top area 1906b. Each of these white keys may be viewed in general 

terms as the combination of two adjoined areas: one forward area rectangle of dimensions 
0.75" by 1.75" and a rear area bar typically at least 0.375" wide (varying with different 
styles of keys). One example black key has a taper leading to a top surface 1906b of 0.25" 
wide by 2" deep, and another example black key has a lesser taper with a top surface 1906b 
10 hat is 0.3125" wide by 2" deep. As one example, a sensor size of 0.25" wide by 0.58" deep 

could be tiled on the six keys 1901 - 1906 according to the arrangement of 1910. As another 
example, a sensor size of 0.25" wide by 0.75" deep could be tiled on the six keys 1901 - 
1906 according to the arrangement of 1920. As a third example, a sensor size of 0.75" deep 
could be tiled on the six keys 1901 - 1906 according to the arrangement of 1930. 

A point not discussed yet - though relevant to all the controllers - is one that is especially 
relevant to all forms of multi-parameter touch-sensing keys: that is the perceptual trade-off 
between note duration and the perception of timbre detail. In short duration notes the ear is 
not able to gather much information about the timbre of the note, while in long notes the 
ear typically examines the timbre, as well as any inherent harmonic animation therein, in 
considerable detail and becomes easily turned away when there is no variation, or easily 
learned predictable variation, in harmonic content over time. The multi-parameter touch- 
sensing keyboard is thus well-targeted for this phenomenon in hearing. On rapid notes 
multi-parameter touch-sensing keys may actually be played with increasing degrees of tim- 
bre-control arbitrariness, while longer notes may be played with a great deal of timbre and 
amplitude variation. Although two degrees of freedom afford by the Moog key is helpful 
in adding per-voice expression for long-duration notes, the three degrees of freedom pro- 
vided by the aforementioned techniques in practice seems to be a minimal control-dimen- 
sionality threshold for useful musical expression. A venture as to why two parameters are 
not enough could start with the fact that there is great importance in relative volume varia- 
tion between voices — this leaves only one parameter then for timber variation which 
quickly bores the ear; adding another dimension allows for more sophisticated temporal in- 
terplays and variations over time in timbre qualities. Empirical support for this is seen in 
the fact that discussions of "timbre space" and "sound color" in the literature devote a min- 
imum of two-dimensions to timbre. A venture as to why the interplay of two timbre dimen- 
sions itself is a minimal control-dimensionality threshold for timbre could resort to an 
abstraction of Figure 17: human hearing is attuned to speech which is in turn a sequence of 
phonemes — each phoneme, roughly, is a vowel sound modulated in time according to 
some consonant aspects and supplemented by, loosely, "percussive" effects in other conso- 
nant aspects. With respect to timbre in phonemes, the ability of human hearing to follow 
and distinguish vowels and their modulation (including diphthongs) is largely centered on 
essentially independent variation in the two formats. Thus, it could be postulated that am- 
plitude variation together with two-dimensional timbre variation engages the speech center 
of the brain in a full and natural way. As with speech, words and phonemes spoken quickly 
cannot be discerned with nearly as much expression as words and phonemes spoken in long 
duration can, and in long duration phonemes the ear is pleased with expressive fluctuations 
in timbre and amplitude. 
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Finally, as to the handling of unisons (and the related problem of melodic line pitch cross- 
ings of uncommon timbre), in usual practice (and prior to the invention) these are typically 
addressed by use of multiple keyboards or by a split of keyboard ranges into independently 
interpreted zones. The addition of proximate keyboards and superimposed keyboards as af- 
5 forded by the invention significantly enhances the practical extent to which and ease by 

which unisons and melodic line pitch crossings may be handled. As a simple example, if 
all melodic lines have timbre ranges that lie in a common range, and unisons sharing the 
same timbre unisons may be naturally handled by superimposed keyboard aspect of the in- 
vention — push the key deeper, or harder, for two (or sequentially, three, four, etc.) notes 
1 0 in unison all following the same timbre control. More generally, proximate keyboards may 

be used to partition the notes that may be played with one hand between two, and in some 
cases three, distinct keyboards; this freely allows the player, in all but some pathological 
cases, to independently control unisons and melodic line pitch crossings without constraint 
as to relative timbre differences. 
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2.1 .7 Video Cameras and other optically-controlled sensors 



Video cameras and other optically-controlled sensors may also be used as control elements 
p within an instrument 100. As with other instrument elements, video cameras and other op- 

=y tically-controlled sensors may be used stand-alone, in arrays, or as component/addendum 

20 to other instruments. Video cameras are especially interesting as controllers because of 

^ available image processing, image recognition, and image motion tracking utilities which 

* have been developed for manufacturing inspection, medicine, and motion- video compres- 

sion together with the ability to actually display a real-time image in recording or perfor- 
mance. 
23 

] n 2.1.7.1 Non-video optically-controlled sensors 

I Tj So as to devote most of the discussion to video, the case of simple non- video optically-con- 

trolled sensors is first considered. A simple example is a set of photo-detectors which are 
=0 used to discretely trigger one or more note, lighting, or special effect events. For example, 

B§ a light-harp without strings may trigger notes, potentially together with selected stage lights 

H and artificial fog blasts, as the fingers interrupt light beams directed towards the photo-de- 

tectors. Another example is that of a stage area with an array of light beams directed to- 
wards an associated first group of photo-detectors: the beams to individual photo-detectors 
of this first group may be interrupted, or redirected by means of reflective surfaces to a sec- 
35 ond group of photo-detectors, by dancers, actors, or musicians in choreographed move- 

ment; the various deactivations and activations of photo-detectors, respectively, may 
trigger one or more of: note, lighting events, or special effects. It is noted that a later de- 
scribed aspect of the invention provides for the generation of an event base on the detection 
of predefined sequences of events; here then certain note phrases or paths through the stage 
40 installation would trigger additional events such as fog blasts illuminated by selected colors 

of light which are distinguished by the pattern detected. 

A more sophisticated use of simple non- video optically-controlled sensors is to continuous- 
ly control one or more of sound, lighting, or special effect parameters; here the photo-de- 
tection is not one of on/off on a relatively narrow beam but rather continuous intensity 
variation of a relatively wider light beam. The light intensity directed at a photo-detector 
may be varied by means of varying the percentage of light interruption by the parts of the 
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human body, clothing, artificial fog clouds affected by a performer, or other translucent, 
light-reflective or light-refractive objects manipulated by a performer. 

In the above, the source light may exist in an environment of performance stage lighting or 
other illumination. To limit interference on the instrument, light sources may be any one or 
more of frequency-modulated, selected-wavelength operation, or minimum-intensity oper- 
ation (via inexpensive low-power lasers) methods. Alternatively, or in addition, a photo- 
detector may be provided with an optically-directional shroud to limit interfering ambient 
light. 

10 It is also possible to actually use stage lights as light sources for photo-detection as an as- 

pect of the invention. For example, a spotlight beam may be directed, via light-reflective or 
light-refractive elements operated by performers, on to one or more photo-detectors oper- 
ating in either discrete-trigger or continuous-variation modes. 

Finally, it is possible for the photo detectors to be color sensitive. This may be done any 
1 5 number of ways, ranging from putting color filters over photo-detectors to using color elec- 

tronic cameras and simple image processing to derive average measured color. Should a 
camera be used for color or other photo-detection roles, photo-detector sites may actually 
y be fiber optic paths that lead to a centralized camera element. Light color directed to the 

photo-detectors may be varied by performers by means of filters, prisms, or other manipu- 
|5 lable translucent, reflective, of refractive objects. 

^ 2.1.7.2 Video cameras 

S Video cameras may be attached to an instrument for showing close-up of the performer's 

R playing. The video close-up feed may be displayed on monitors during a performance or 

?5 recorded, and as discussed later, potentially involving other video sources and potentially 

O with or without special effects. For movable instruments, such as guitars, woodwinds, etc. 

\u this can create an interesting visual effect as the instrument profile will be firmly fixed in 

1= the video image while the ambient visual background will move as the performer moves 

•y the instrument. These visual effects seem to work best with instruments that have sufficient 

physical inertia and/or which are supported by straps; instruments subject to significant un- 
~ s ~ damped motion, such as flutes, may actually have so much background motion that the im- 

age is uncomfortable to watch. 

Video cameras, be they attached to an instrument or not, may also be used as instrument 
2$ elements by processing the video image signals to degrees that range from simple average 

image brightness calculation through pattern recognition to image interpretation. In a sim- 
ple example, the luminance signal for each video frame or interlace-field (i.e., only the odd 
or only even lines) may be sent to an integrater element followed by a sample-hold element; 
the integrator may be further enhanced to not integrate during retrace intervals. The result 
40 gives the average brightness of the processed image. Adding two such additional inte- 

grate/hold elements and feeding the three the red/green/blue decomposition of a color video 
signal makes an image-averaged color detector. In these ways the same camera that pro- 
duces performance and/or recording video images may be used as a non- video optical sen- 
sor in the manners described earlier. This primitive capability, then, may allow a performer 
45 to tilt or rotate the instrument 100 position so as to include stage lights or background im- 

ages of particular brightness and/or colors, direct or impede incoming light with the hand 
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or objects, cover the lens, etc., and in so doing trigger and/or continuously control sound, 
lighting, or special effect events. The latter may occur when the video image is being dis- 
played and/or recorded or with the video signal used solely in an instrument mode. 

Far more valuable is the use of the spatial capture aspects of a video camera. A simple ex- 
ample of this would be to split the image into "sub-image cells" (i.e., half, quarters, etc. of 
the entire video image) using various means and again deriving average luminance and/or 
color information from each of the cells. For small numbers of cells this may be done with 
analog electronics: sync detectors trigger one-shots that gate specific integrate/hold circuits 
for specific intervals of horizontal scan lines in specific vertical regions of the image. Dig- 
ital methods may also be used, for example: reading the image into a frame buffer which is 
then analyzed in the retrace interval for the next frame, doing running calculations on the 
video signal as the fields are scanned, etc. Digital methods will typically scale to higher res- 
olutions and more complex functionalities and thus in many cases may be preferred. Digital 
methods may be implemented with special dedicated hardware or standard personal com- 
puters fitted with standard video capture and MIDI interface cards, etc. Such personal com- 
puter implementations may implement a number of image processing, parameter 
derivation, and control signal assignments in a flow virtually identical to that of Figure 13. 
These functions may be done in software run on the personal computer or in part or in 
whole by dedicated hardware boards or peripherals (for functions such as video acquisition, 
pattern recognition, etc.) 

With the ability to process images at higher resolutions and in more complex ways, it be- 
comes possible to use video in increasingly valuable ways as an instrument element. By 
correlating higher resolution image area measurements, it becomes possible to recognize 
patterns and shapes and derive parameters from them in real-time. In fact, the same image 
processing software structures used in pressure-sensor array touch-pads, or even exact por- 
tions of software itself, may also be used to process video images in real-time, replacing 
pressure pixel information with, for example, luminance pixel information. These algo- 
rithms may be enhanced further by exploiting available color information as well. The 
shapes recognized and some of the parameters derived from them are likely to have a some- 
what different quality: the 3D-projected-to-2D nature of camera images, gradients of lumi- 
nance created by shadows and reflections, as well as the types and (potentially) ranges of 
shapes to be recognized typically differ significantly from those discussed in the pressure- 
sensor array touch-pad context. Nevertheless, similar software structures may be used to 
great value. Specific types of shapes and patterns - such as written characters, particular 
gradients in brightness or color, separation distances between bars and/or bar widths - may 
be particularly useful variations from those shapes and patterns discussed in the context of 
pressure-sensor array touch-pads. 

Next: examples of how video cameras supplemented with these capabilities may be used to 
trigger events and/or continuously control sound, light, and special effects. 

A first example is that of recognizing the human hand posture, position, and proximity to 
the camera in 3 -space. Simple hand orientation and posture geometry may be used to create 
specific control signals. In a more advanced implementation, dynamic gestures may be rec- 
ognized. These two capabilities give the system, with sufficient software, the ability to rec- 
ognize a few if not many verbal hand signals; with yet more enhancements, potentially 
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including the ability to recognize the roles of two hands with respect to the human body, 
the recognition capabilities could include, for example, formal ASL as well as particular 
dance postures. The ability to recognize postures of hand, hand/arm, hand/arm/body, etc. 
allows hands, dance, "conducting" (not necessarily restricted to formal conducting ges- 
tures), etc. to be used directly for the control of sound, lighting, and special effects. 

In another class of examples, video cameras may recognize, and derive parameters from, 
characters and/or patterns available on a stage. Such characters and/or patterns may be 
brought before the camera, exposed and obfuscated from the camera; the camera may be 
turned towards the characters and/or patterns, etc., resulting in derived parameters and is- 
sued control signals. Stage cameras may also be used to recognize and track the location 
and some aspects of body orientation and posture of performers, deriving parameters and 
issuing control signals from these as well. 

In each of the above examples, it is noted that the use of two or more cameras, either in 
stereoscopic layout similar to those of human eyes or in an orthogonal layout (i.e., forward 
facing camera and overhead camera covering the same 3 -space region), may be used to re- 
solve 3D-to-2D projection singularities in the pattern and shape recognition and process- 
ing. 

As a third class of example, recent developments have allowed for the recognition of hu- 
man facial expressions from video images and even degrees of lip reading. These recogni- 
tion and parameter derivation methods may also be adapted in the invention to provide the 
ability for the human face to be used as a controller for sound, lighting, and special effects. 
Simplified systems can be created to recognized and parameterize a few selected expres- 
sions or to recognize and measure geometric variations in specific areas of the face. 

From a formal, traditional music perspective, much of the above may appear to be gim- 
mickry with meaningful application at best in avant garde installations or modern play 
products. In one response to this, directed on hand posture capture, it is noted that the hand 
in 3-space is clearly the most physically expressive aspect of the human body and is used 
to control almost all musical instruments but by very restricted geometric means. Freeing 
the hand to move unrestricted allows considerably more expression to be captured. Further 
then, as a fourth example, advances in cost reductions for video cameras and signal pro- 
cessing can make it possible for an array of cameras to be devoted to a traditional instru- 
ment controller, such as a keyboard, drum head, or flute key array (as well as, for example, 
a pressure-sensor array touch-pad) so as to capture hand expressions that cannot otherwise 
be cost-effectively captured from the instrument controller. 

Final, a brief preliminary discussion is provided here on the significant role of video in 
compositional and performance semiotics. For many years music, dance, art, film, plays, 
literature, poetry, linguistics, and other fields have come under study and compositional 
methods involving common abstractions or "signs" that live within and among their works 
and idioms. More will be said later about the invention as a whole as an environment for 
more significantly exploiting semiotics as a compositional and performance tool. However, 
video cameras used as an instrument element, either with or without the video stream being 
displayed or recorded, offer a special role in the creation of semiotic elements because they 
may be used to link visual symbols of object and body to sound, lighting, and special effects 
which in turn may have assigned and/or intrinsic semiotic content. 
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2.1.8 Singing and Sp ch D tection, R cogniti n, and Paramet rization 

Speech recognition systems have become increasing accurate and inexpensive. These tech- 
nologies can, in many valuable ways, be adapted to also recognize sung words and/or pho- 
nemes. Recognized words or phonemes may be used to trigger any of sound, lighting, or 
special effect events, while existing pitch detection and amplitude following technologies 
(as found, for example, in the early Roland CP-40 product or in the more modern MidiVox 
SynchroVoice product) may be used to derive continuous control signals. In addition, inter- 
event timers may be used to measure individual word and/or phoneme duration. 

1 o These singing and speech recognition capabilities together with their parameterization also 

have significant potential value in the aforementioned creation of semiotic elements be- 
cause they can be used to link verbal linguistic events and expression to sound, lighting, 
and special effects which in turn may have assigned and/or intrinsic semiotic content. 

Air Pressure, Air flow, and Air Turbulence Sensors and Transducers 

Air flow, or "breath," controllers for musical instruments are known and have been em- 
ployed in electronic woodwind-like controllers. It is a provision of the invention to include 
these along with air pressure and air turbulence sensors and transducers as elements of an 
instrument entity 100. In particular, air pressure-sensors can be attached to air bladders to 
form a particular kind of pressure or squeezing controller. Air pressure- sensors can also be 
introduced into a wind instrument interior in an instrument where subsonic variations in 
ambient pressure occur as the instrument is played. 

Traditional wind instrument players often invoke air turbulence effects, such as transient 
"chiffs", tongue trills, overblowing, etc. Air turbulence is then also a candidate control in- 
terface for use in an electronic instrument entity 100. Air turbulence sensors may be crafted 
in various ways, including by means of signal processing the output of any one or more of 
air flow and/or air pressure-sensors. A simple example would be to define a high-pass cut- 
off frequency for air flow and/or air pressure variations and another (higher) low-pass cut- 
off frequency for the lowest musical "pitched" frequencies; the energy in the remaining 
band of frequencies would be a crude measure or air turbulence. In a more sophisticated 
implementation, an array of air pressure-sensors can be distributed throughout a wind tube 
and sensor-array signal processing techniques can be used to separate turbulence signals 
from environmental acoustic noise, standing waves in the tube, etc. 

35 2.1.10 Clothing, Jewelry, Skin, and Muscle Sensors 

Sensors on the human body have been used in some dance performances to control sounds. 
The invention provides a generalization of this for synergistic use in conjunction with oth- 
ers of its aspects. 

40 Sensors may be attached to the human body by means of clothing, jewelry, straps, adhesive 

pads, etc. These sensors can be of a variety of types: position, motion, optical, skin resis- 
tance, muscle activity, etc. and may be used to capture body position, posture, activity, en- 
vironment, etc. and convert these into control signals used to control sound, lighting, and 
special effects. Sequences of control signals can also be interpreted as gestures by recogni- 

45 tion systems which in turn can be used to generate yet other control signals. Interfaces to 

the sensors, taken collectively as an instrument entity 100, to one or more signal routing, 
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processing and synthesis entities 120, may be done by means of radio, wireless optical, fi- 
ber optic cable, electrical cable, or combinations or sequences of these. 

Although the sensors described here taken as an instrument entity 100 may be used in iso- 
lation, there is particular synergistic value in using these in conjunction with other instru- 
ment entities in a performance or recording situation. For example, a particular body 
motion or gesture (such as raising an arm, swinging a hand, jumping, etc.) may have sig- 
nificant artistic value at a critical moment but not be captured by another instrument entity. 
As another example, in recording sketches during a composition phase, particular body mo- 
tions or gestures can be used to call attention to specific aspects of the sketch for future re- 
view. 

2.1.11 Stage Environment and Macro-Environment Sensors 

Sensors other than optical can be distributed on a stage and/or on component installations 
^ on the stage (for example staircases, risers, scaffolds, sculptures, props, etc.). Sensors can 

also be used to measure larger environments ranging from audience activity to outdoor me- 
teorology. The sensors can include proximity, position, motion, weight, temperature, hu- 
midity, etc. and can be used to create control signals. As a result, these arrangements can 
^ be formalized into an instrument entity 100. 

i%Q Examples of such usage include human proximity and/or interaction with props or sculp- 

M tures, tracking of artificial fog cloud migration across a stage, detecting the location of per- 

ry formers on staircases or risers, detecting audience motion activity, characterizing room- 

s H internal and room-external meteorology (such as wind speed, wind direction, rainfall, wind 

Jz! and/or rainfall noise, etc.) to bring it into an aspect of the performance. 

% 

2.2 Vibrating-Element Instrument Elements and Subsystems 
Y: 2.2.1 Single-channel audio signal handling 

=u The invention provides for the inclusion of traditional group (or "composite") audio signals 

OX) such as a group pickup serving all strings on a traditional electric guitar. These can be treat- 

i^' ed as a peer to any of the multi-channel audio signals or of special significance because of 

its timbre, functionality, or traditional use. As will be illustrated in the discussion of layered 
signal processing, such a signal can be processed so as to create the subtle or dominate 
backdrop against which processed multi-channel signals are superimposed. In some situa- 
35 tions, multi-channel signals on the instrument may be combined to create a single channel 

audio output, as in the case where individual piezo bridge pickups are only one of a plural- 
ity of multi-channel signal sources on an instrument; simple full or partial mix-downs may 
be provided for use when such multichannel sources are not featured in a multi-channel 
manner so as to conserve channel usage on the generalized interface 110. This can be par- 
40 ticularly valuable in complex instruments with many arrays of vibrating elements such as 

those in Figures 4-5 and many others to be discussed. 

2.2.2 Multi-channel audio signal handling 

The use of various types of musically-oriented signal processing with electronic stringed 
45 instruments has been common in popular music almost as long as there have been electron- 

ic stringed instruments. Typically a single pickup is used to capture audio signals from all 
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vibrating elements on the instrument (although there may be a plurality of such group pick- 
ups on a given instrument so as to obtain different selections of timbre). 

The invention provides for the use of multi-channel electric transducer arrangements, by 
which each vibrating element (string, tyne, membrane, etc.) of an electronic instrument 
with multiple vibrating elements is provided with an independent isolated electrical output, 
and dedicated signal processing can be applied to the signal of each vibrating element or 
incomplete combinations thereof, to achieve significantly important musical functions — all 
done in a way where the same interfaces, multi-channel signal routing and processing, and 
internal instrument electronics can be reused across a variety of instruments. 

Multi-channel vibrating element pickup arrangements, by which each vibrating element 
(string, tyne, membrane, etc.) of an electronic instrument with multiple vibrating elements 
is provided with an independent isolated electrical output, have been commercially avail- 
able but in largely hidden forms, most commonly used in synthesizer interfaces for guitars. 
Beyond such synthesizer interfaces, and the recent Roland VG- 1 product discussed later, 
the usage of such multi-channel vibrating element pickups has been limited to roles in vol- 
ume equalization and imaging in a stereo sound field on only a very few electric guitars 
models. Such musically-oriented signal processing is only known to have been applied to 
the summed mixture of all vibrating elements of the instrument, not for individual or sub- 
groups of the vibrating elements of the instrument. 

Conventional signal processing can be used on each vibrating element signal to create "gen- 
eralized pedal steel guitars" (augmenting or replacing mechanical pedal tuning changers), 
instantly retunable guitars (augmenting or replacing mechanical tuning changers such as 
the Hip-shot "Trilogy"), multi-modal Indian sitars (where drone and sympathetic strings 
can be electronically retimed while playing, allowing a more robust mix between Eastern 
and Western tonality in musical form), spatially animated instruments where individual vi- 
brating element sounds are location modulated within a stereophonic or other spatial sound 
field, and mixed timbre instruments where different signal processing methods are applied 
to each string. 

Standard pickup elements available to implement individual pickups for each vibrating el- 
ement include piezo contact elements, installed on a bridge acoustically isolated from other 
vibrating elements, and non-contacting coil-based electromagnetic pickup elements. Opti- 
cal pickup products have also been devised, and a coil-less Hall-effect pickup method has 
been taught as U.S. patent 4,182,213. Both optical and Hall-effect methods do not involve 
contact with the vibrating element. Figure 20 shows electromagnetic, Hall-effect, piezo, 
and optical pickup methods for deriving separate audio signals for each vibrating element 
of a multiple vibrating element instrument entity. Electromagnetic coils and Hall effect el- 
ements require the string material to be ferromagnetic while piezo and optical methods do 
not. 

It is noted that a pickup localized for individual vibrating element must by its nature have 
small geometry. For the pickup technologies not involving contact with the string (e.g., 
electromagnetic coils, Hall effect, and optical) multiple small pickups can be aligned along 
a vibrating element's length; the resulting multichannel signal may be handled with multi- 
channel signal processing, selected by a switch, selectively mixed/ morphed, etc. to obtain 
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a range of tones. In one implementation the selection, mixing, morphing of the pickup sig- 
nals, and hence the resulting output tone, may be operated by control signals. 

It is noted that excessive magnetic fields from a large number of magnetic pickups may 
make a low-mass vibrating element such as a thin string vibration go inharmonic. Although 
this should be a design consideration with a number of pickups, it can also be used to pro- 
duce special effects. The invention thus provides that one or more electromagnetic coils, 
which may or may not otherwise double as pickups, be used to issue localized DC magnetic 
fields of varying intensity for inducing inharmonic effects on one or more selected strings, 
most advantageously under control signal control. The coils may create the DC magnetic 
fields themselves or instead cause a permanent magnet to vary its distance to the vibrating 
element via solenoid structures. 

The sloped bridges of sitars and other twanging/buzzing stringed Indian instruments have 
not to date lent themselves to individual piezo bridge structures. This is not impossible; the 
15 invention provides for individual miniature sloped bridges, one for each string, to be em- 

bedded with its own piezo pickup element. Such bridges can also be used with non-string 
vibrating elements, such as bars and tynes, to create new types of sounds. This method can 
13 also be adapted to the very gradual and softer sloped body contact of certain African harps 

;p whose strings buzz against a typically animal fur-covered harp body. Alternatively, Figure 

2§ 21 shows how an off-bridge buzz-plate, such as those provided by Biax to simulate a fret- 

less based sound with a conventional fretted bass, may be combined with a piezo bridge 
sensor in replacement of a gradient buzz-bridge so as to permit the use of non ferromagnetic 
strings. 

Vibrating Element Excitation 

The use of "controlled (acoustic) feedback" with electronic stringed instruments has been 
in common use in popular music since at least the 1960's. It has been possible to replace 
the acoustic excitation of string resonance with electromagnetic excitation (as embodied by 
the Heet Sound E-bow) for some time, but only for one string at a time and via hand-held 
mechanically operated apparatus. The practice of electromagnetic excitation in non- 
stringed musical instruments with vibrating elements is not currently known. 

The invention presents a system using electromagnetic excitation of the vibrating elements 
of an electronic instrument to produce controlled feedback relationships with signal pro- 
cessing control of the feedback characteristics, typically hands-free as desired, with either 
standard parts (for inexpensive mass manufacture and retrofit) or more specialized parts (to 
provide additional features). 

Figure 22 shows the basic idea of controlled feedback as used in recent contemporary music 
circa 1960. A vibrating element 2201 within the instrument is coupled (by electromagnetic, 
40 optical, or mechanical means) to an electrical transducer 2202 (electromagnetic, optical, 

Hall-effect, piezo, etc.) which converts the vibration to an electrical signal 2203. The elec- 
trical signal 2203 is applied to a power amplifier 2204 which drives a loudspeaker 2205 
which is acoustically coupled 2206 (by means of air, mounting apparatus, etc.) to the vi- 
brating element 220 1. This creates a feedback arrangement allowing vibrations of specific 
45 frequencies to resonate within the resulting closed-loop system. By exciting or damping the 

vibrations of the vibrating element 2201, changing the characteristics of the frequency 
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and/or phase response of the speaker's power amplifier 2204 \ and/or changing the charac- 
teristics of the means of acoustic coupling 2206 (as in changing the distance between the 
vibrating element 2201 and speaker 2205), the "controlled feedback" methods used in pop- 
ular music are obtained. Note, however, that the required acoustic coupling characteristics 
are affected by factors such as volume level (typically this must be relatively high), speak- 
er/room geometry, room acoustics, and other difficult to control factors that can often be 
unpredictable liabilities. 

The invention provides for an approach to replacing the acoustic excitation component of 
this process with electromagnetic excitation. Figure 23 shows an example implementation 
of simple approach for replacing acoustic excitation of a vibrating element with electro- 
magnetic excitation. In this case the vibrating element 2201 must be ferromagnetic, al- 
though the transducer need not use ferromagnetic means itself. Here, rather, the acoustic 
coupling 2206 is replaced by electromagnetic coupling 2209, produced by an electromag- 
netic coil 2208 with an internal magnet or other magnetic bias that replaces the speaker 
2205. Without this nominal magnetic field, the string will be excited with a full -wave rec- 
tification as the ferromagnetic string is drawn to the coil regardless of the direction of cur- 
rent flow. In this configuration, the power amplifier must now match the coil's electrical 
drive requirements which can differ from the speaker's, hence the speaker's power ampli- 
fier 2204 is replaced by the coil's power amplifier 2207. A signal output 2210 for subse- 
quent amplification and signal processing can be taken off at the transducer. (In particular, 
this output arrangement 2210 differs from the Heet Sound E-bow where no such output is 
provided or relevant for that product). The result is a system that provides a comparable ar- 
rangement to that of Figure 22 without the requirements of high volume level, speak- 
er/room geometry, room acoustics, and other liabilities of acoustic coupling. 

q It is also noted that as piezo elements both convert vibrations into alternating current signals 

iy and, reciprocally, convert alternating current signals into mechanical vibrations, a piezo 

!= group element bridge pickup can be used, in lieu of a coil, either as the audio signal pickup 

Q or as a mechanical drive exciting element. Further, the signal pickup can also be optical or 

p0 Hall effect. If both the signal and drive elements are electromagnetic (coils or Hall for sig- 

" nal pickup, coil for drive) undesirable magnetic coupling, not unlike that of an electric 

transformer, can occur. This effect may be minimized if said signal and drive elements are 
sufficently separated and/or shielded or otherwise localized (for example, with a two- 
coil/opposite-magnet arrangement. 

Figure 24 shows various combinations of piezo and electromagnetic vibrating element 
pickups and exciter drivers for separately controllable excitation of each vibrating element. 
In one arrangement, the string 2400 suspended over the bridge 2401 is electromagnetically 
coupled to two electromagnetic coil pairs 2411 and 2412. Each coil pair is in a standard 
"humbucking" arrangement with complementary magnet pole directions 2413a, 2413b and 
complementary winding directions 2414a, 2414b so as to significantly localize the magnet- 
ic coupling region about the coil. Here either coil may be used as the signal source or as the 
driver. The source coil can also be replaced with an optical or Hall-effect pickup. In a piezo- 
based arrangement, the string 2400 is in contact with a piezo pickup element 2421 on the 
bridge 2401, and the string 2400 is magnetically coupled with an electromagnetic coil pair 
^ 2422. Here either the coil pair or the piezo may serve as the driver and the other serve as 
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the signal source. In cases where the piezo element acts as the driver, the electromagnetic 
signal source element may be replaced with a Hall effect or optical technology pickup. 

It is noted that the invention provides for the above discussions to apply equivalently should 
the signal source and driver elements serve an individual vibrating element or a group of 
vibrating elements. The invention also provides for the case where either the signal source 
or driver is a single element unit while the other is a group element unit; such configurations 
are easily supported by the signal routing, processing, and synthesis entity 120 (referring to 
Figure 1). In particular the invention provides for the case where a multiple vibrating ele- 
ment instrument has at least one of the following: individual source and individual excita- 
tion, group source and group excitation, individual source and group excitation, and/or 
group source and individual excitation. 

Since the driving element (coil or bridge piezo) may be mounted in permanent relation to 
the vibrating element, it is possible to replace conventional means of altering the acoustic 
coupling with electronic signal processing means 221 1. Figure 25 shows adding signal pro- 
cessing for spectral and amplitude control of electromagnetic excitation. For example, 
fixed or adaptive equalizers can be used to alter the frequency and phase response of the 
signal/vibration/transducer loop, permitting additional control over which vibrational har- 
monic^) are emphasized in the feedback. Attenuation can be used to vary the degree of 
feedback. Delay can be used to alter the attack characteristics of the resonance behavior. 
Dynamic compressors and expanders can be used to vary the ease and dynamics of the res- 
onance behavior. Many interesting special effects are possible, such as using pitch-shifters 
alone or in combination with delays to transfer energy between vibrational modes. In the 
case of a drive coil, a pitch shifter or octave divider (such as an inexpensive toggle flip-flop) 
may be used to create a drive signal that is an octave lower than the string signal and thus 
eliminate the need for a magnet in the drive coil. The invention provides for any driver elec- 
tronics and/or signal processing to be any of the following: internal to the instrument, 
mounted on the outside of the instrument as an add-on module, remotely located off the in- 
strument (particularly in the signal routing, processing, and synthesis entity 100), or any 
combination thereof. 

In most electronic instruments, a single pickup serves many if not all the featured vibrating 
elements. The invention provides for the approaches discussed thus to also be applied to 
such instruments using conventional components. Figure 26 shows an example arrange- 
ment involving multiple vibrating elements served with a group pickup and which are also 
subjected to common electromagnetic excitation using conventional guitar pickup compo- 
nents. (In the example of Figure 26 it is understood that signal processing 2211 may be in- 
troduced or omitted from the feedback loop as appropriate or desired and that a group piezo 
bridge may, in some constructions, serve as a driving element in place of the pickup coil.). 
A plurality of vibrating elements 2201.1 - 220 Ln share a common group pickup transducer 
2202 and a common electromagnetic coil 2208. This arrangement is very simple to imple- 
ment and very useful musically for traditional electric guitarists. A simple prototype can be 
made using an electric guitar with two humbucking-pickups (such as a Gibson ES-335). 
The rear pickup can be used as the transducer 2202 and the front pickup as the coil 2208 
(the humbucking pickups assist in decoupling the coils, decreasing a parasitic "transform- 
er" effect). Almost any power amplifier of sufficiently high enough current or voltage 
drive, (for example, even a Fender Bassman tube guitar amplifier) can be used as the coil's 



WDW 001/01 VS 



power amplifier 2207, directly driving the coil (despite the impedance mismatch) from the 
amplifier speaker output connector. 

Example Electronic Controller Instruments 
Touch-Pad Array 

Touch pad instrument elements, such as null/contact types and pressure-sensor array types 
described earlier, can be used in isolation or arrays to create electronic controller instru- 
ments. The touch-pad(s) may be advantageously supplemented with panel controls such as 
push buttons, sliders, knobs as well as impact sensors for velocity-controlled triggering of 
percussion or pitched note events. In the case of null/contact touch-pads, impact and/or 
pressure-sensors can be added to the back of the pad and the pad suspended in such a way 
that it can be used as an electronic drum head. If one or more of the touch-pads is transpar- 
ent (as in the case of a null/contact touch screen overlay) one or more video, graphics, or 
alphanumeric displays may placed under a given pad or group of pads. 

Figure 27 illustrates examples of single, double, and quadruple touch-pad instruments with 
pads of various sizes and supplemental instrument elements. A single touch-pad could 
serve as the central element of such an instrument, potentially supplemented with panel 
controls such as push buttons, sliders, knobs as well as impact sensors. In Figure 27, a trans- 
parent pad superimposed over a video, graphics, or one or more alphanumeric displays is 
assumed, and specifically shown is a case of underlay graphics cues being displayed for the 
player.Two large sensors can be put side by side to emulate the left-hand/right-hand layout 
of many hand drum arrangements such as tabla/baya, congas, etc. This is particularly suit- 
able for pressure- sensor array touch-pad elements where a larger pad-area (for example 8 
to 12 inches square) could be advantageous for detailed control. Because of the extensive 
capabilities of either type of touch-pad element provided for in the invention, this arrange- 
ment is by no means limited to percussion applications but rather easily serves as a far more 
general purpose left-hand/right-hand multi-parameter controller. In variants of this ar- 
rangement that are intended specifically for tabla/baya emulation, the relative size of the 
two pads and angle of placement with respect to the floor can be arrangement to match that 
expected by an experienced tabla player. Instruments involving arrays of larger numbers of 
touch-pads can also be valuable. Here it may be advantageous to make the pads smaller so 
that the fingers of a single hand can touch two or more pads simultaneously. 

Foot Controllers 

With the extensive real-time control capabilities provided for in the invention, foot control- 
lers can be especially valuable. They can select preset configurations at various points in a 
control hierarchy, issue notes or chords, control timbre, alter lighting, invoke special ef- 
fects, etc. In general a commercially available floor controller typically includes a plurality 
of momentary action foot-switches, and various visual status indicators such as LEDs over 
momentary action foot-switches and a master status (and programming) display. Many 
such products also include provisions for rocker foot pedals to control continuous parame- 
ters, either via external connection (as with the Digitech PMC- 10 and Digital Music 
"Ground Control" products) or internally (as with the ART X-15 product). With the excep- 
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tion of the Digitech PMC- 10, the control assignment and organization capabilities of these 
controller products have historically been quite limited, and as all the products seem aimed 
largely at issuing MIDI program change commands, the number of foot-switches has been 
small. Further, the rocker foot pedals control only one parameter at a time. 

^ The invention provides for extensive elaboration over these products by supporting any of 

multi-dimensional rocker pedals, arbitrary control signal assignment, control signal assign- 
ment organized by selectable pages, separate alphanumeric function display for each foot 
control (switches and pedals), pause operations, and real-time event play-back capabilities. 

10 The traditional way to control volume on an electronic keyboard instrument is by a means 

of a rocking floor-level foot-pedal. More recently such pedals have been used to generate 
continuous-range control signals such as MIDI messages, though allowing the control of 
only one continuous-range parameter at a time. Many years ago a number of "volume/tone" 
foot pedal products were available, though none appear available at this writing. These 

1 5 products offered a rocker capability devoted to controlling instrument volume supplement- 

ed with a left-right twist capability devoted to the control of instrument tone. Such arrange- 
ments may be used to double the number of foot controllable parameters that can be 

□ controlled in roughly the same physical layout area together with the bonus of allowing a 
=y foot to control two continuous-range parameters at once. 

2p Figure 28 illustrates some enhanced foot-pedal arrangements which permit simultaneous 

jy single-foot adjustment of a plurality of continuous range parameters for use with floor con- 

[jz trollers. The use of rocking foot-pedals to control two continuous-range parameters at once 

p may be enhanced by using one or more side-mounted spring-levers. Side-mounted momen- 

ta tary-action switches have been used on rocker foot-pedals for mode control (products by 

3-5 Ernie Ball and Soloton), but side-mounted spring-levers are particularly advantageous for 

□ continuous range parameters that have a specific nominal value. For example, these can be 
5^ used in conjunction with pitch-shifters to modulate pitch as do the foot and knee levers of 
l Z a pedal steel guitar, or in complementary pairs to emulate the action of a synthesizer mod- 
% ulation wheel or an electric guitar vibrato "whammy bar." In Figure 28, one or more side- 

mounted spring-levers 2803, 2804 may mount on either the base 2801 or rocker plate 2802 
of a foot-pedal. A spring lever may directly operate a slide, geared, pulleyed, etc., potenti- 
ometer, an optical sensor, magnetic sensor, pressure-sensor, etc. to produce an electrical 
signal. If two side-mounted spring-levers 2803, 2804 are positioned on opposite sides of 
the pedal, two mutually-exclusive parameter adjustments can be realized (as found in pedal 
35 steel knee-lever pairs and in the action of a synthesizer modulation wheel or an electric gui- 

tar vibrato whammy bar). It is also possible to mount a springed center-return synthesizer 
modulation wheel 2805 at the far end of the rocker plate if the arrangement and materials 
used forego breakage in heavy usage situations. 

, 4Q Further, it is possible to add a third control continuous-range adjustment capability on the 

rocker pedal by measuring the length-axis rotation of the foot: this could be done by various 
methods. As one example, a two-dimensional "volume/tone" foot pedal with control mo- 
tions up-down 2810a and twisting 2810b may be modified to permit length-axis rotation of 
the foot 2810c and measure it with a potentiometer or sensor. Another method would in- 

45 volve putting at least two pressure-sensors 2813 on the twist plate 2812 of a non-modified 

two-dimensional foot pedal 2811 and deriving a control signal from these. A third way 
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would be to mount a springed center-return synthesizer modulation wheel at the far end of 
the twist plate if the arrangement and materials used forego breakage in heavy usage situ- 
ations. Other methods can be used for multi-dimensional foot controllers, such as the 
null/contact touch-pad and pressure-sensor array touch-pad elements discussed earlier 
which can be adapted for foot operation. 

The invention provides for arbitrary assignment of control signals to specific foot- switches, 
foot-pedals, and other foot controllers. As an example, one or more MIDI messages could 
be assigned to each foot-switch, foot-pedal, or other foot controller as is largely done in the 
Digitech PMC- 1 0 and with other functionality as the custom message construction and hi- 
erarchical ganging provided by, for example, the Peavey PC- 1600 slider/button controller). 
A particularly valuable additional function would be that of issuing continuous controller 
messages that oppositely complement the basic control signal value: for example, in MIDI 
messages where "Continuous Controller" control values lie in the range 0 to 127, if a con- 
tinuous foot-pedal position causes a first control signal to be issued with value of "x", it is 
also possible to enable the subsequent transmission of a second separate control signal to 
be issued essentially simultaneously with a value determined by the algebraic relation 
"127-x"; such complementary signals may be used for many purposes, for example prorat- 
ing an audio mix between two sources, prorating modulation indices among two synthesiz- 
er voices, etc. 

Stored program memory may be used to retain these assignments. In this situation it is ad- 
vantageous to allow for multiple stored program selections to be recalled, thus allowing for 
multiple assignment sets for each foot-switch, foot-pedal, etc. Each assignment set could 
be thought of as a "page." Pages could be copied as a whole and edited. These capabilities 
would be similar to those of the Digitech PMC- 1 0 and Peavey PC- 1 600 products. However, 
because of the number of controller assignments and the diversity of possibilities it is de- 
sirable to add physically adjacent to each foot controller an alpha-numeric display indicat- 
ing the current assignment and status of that controller: in particular, for each given selected 
page, each controller display may show one or more of the currently assigned function, the 
current value(s) transmitted or last-transmitted, any additional identifying information such 
as short-hand names or relationships with other controllers, etc. LEDs may be provided for 
quick reference as to which foot-switch and which continuous foot controller (pedal, touch- 
pad, etc.) were last operated; as an enhancement these LEDs could be bi-color and of the 
two LEDs lit at a given instant (one for last foot-switch used, the other for continuous con- 
troller used), one color (i.e., green) could be used to indicate to overall last operation while 
a second color (i.e., red) would be used to illuminate the remaining lit LED. 

The operation of a foot-switch may be assigned, under stored program control, to issue one 
or more simultaneous control signals, or short burst of contiguously-sequential control sig- 
nals such as a group of MIDI messages. These control signal events may occur on the de- 
pression of the foot-switch, its release, or both. The foot switch may also be configured to 
operate in a toggle mode using a divide-by-two counter and messages can be issued on each 
toggle transition. These useful features can be found on, for example the Digitech PMC-1 0, 
but a number of useful enhancements are provided for by the invention. One enhancement 
would be to allow any specific pedal to independently operate in a generalization of toggle 
mode to permit a round robin selection of 3 or more states (for example "off," "slow," "me- 
dium," "fast"). Another enhancement is to allow a more complicated state transition map 
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involving a group of foot-switches. Yet another enhancement is to permit timed events to 
be issued. The simplest of these would be timed pause operations between control signal 
events, while a more enhanced implementation would permit real-time control event play- 
back capabilities to be assigned to a foot-switch. Such real-time event sequences could in- 
clude not only note sequences but also trajectories of continuous parameters (for example, 
exponential transients or linear ramps). Further, the invention provides for the issuance of 
the same selection of possible control signal options upon incoming or outgoing page- 
change events during a stored memory page change. 

Finally, larger foot controller assemblies with appropriate organizational and ergonomic 
layout are advantageously provided for by the invention. Among the factors here are overall 
ergonomic operation, putting some foot controlled elements closer to the user for fast or 
intimate use with others farther away for background or occasional use, and an overall 
physical and operational organizational hierarchy. In implementing such hierarchies, each 
full stored program page can involve one or more sub-pages which also be used as a part 
of other full stored program page. Although such a sub-page can in general be assigned to 
any foot operated control element, it typically would be useful to confine each sub-page to 
a pre-defined reusable geometric region in the overall foot controller layout. Further, the 
invention provides for sub-pages to be changes within an active full page. 

Figure 29 shows some example layouts involving 2 geometric regions for a moderate num- 
ber of foot operated controllers 2920 and 4 geometric regions for a larger number of foot 
operated controllers 2940. The smaller arrangement 2920 features a rocker pedal with two 
side-mount spring controllers 2903 and two rock/twist pedals 2900 as well as two geomet- 
ric regions — one proximate 2921, another remote 2922 — of foot-switches 2905. Each 
foot-switch and pedal is provided with an alphanumeric display 2906 and a last-operation 
indicating-LED 2907. The larger arrangement 2940 features an advantageous layout of two 
proximate geometric regions 2941, 2942 of foot- switches, two increasingly remote geo- 
metric regions 2943, 2944 of foot-switches, one proximate rock/twist pedal 2900, remote 
pedals of rocker only 2901, single side spring lever 2902, and double side spring lever 
2903, as well as a foot or toe operated touch-pad 2904. Each foot operated controller is also 
provided with an alphanumeric display 2906 and a last-operation indicating-LED 2907. 
The layout used in the larger unit permits logical association of groups of switches and ped- 
als in a wide variety of contexts. In either the smaller or larger arrangement, the more re- 
mote controllers can be put on progressively higher risers to create a staircase layout. These 
arrangements permit for an arbitrary logical hierarchy of page and sub-page recall control 
and arbitrary assignment of which buttons may be used to do this. In some cases it may be 
desirable to have an additional summary display showing the selected page and sub-page 
status in one location at a glance. 

Multi-Tier Proximate/Superimposed Keyboards 

The proximate and superimposed keyboard elements described earlier can be combined to 
create a powerful enhanced keyboard controller. In an example implementation, an ar- 
rangement of three proximate keyboards such as shown in Figure 3 may be brought togeth- 
er in a common unit. This unit may also advantageously include one or more of any of 
sliders, knobs buttons, joysticks, touch-pads, strum-pads, impact sensors, etc. Further, it is 
noted that any of the keyboards here may be either of a standard variety or any of the more 
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advanced keyboards described later (miniature, superimposed, multi-parameter keys, pres- 
sure-sensor array, etc.). It is also noted that this technique may be applied to other types of 
keyboards with applicable types of key geometry. 

One-Hand Enhanced-Drum-Roll Controllers 

The invention provides for one-handed methods of performing drum-rolls with some ad- 
vanced capabilities. The basis of the method involves the proximate location of two elec- 
tronic impact sensors and/or touch pads oriented to be facing each other, but the method 
can also be used with acoustic drums. The arrangement can be small in scale, i.e., played 
with a single finger, or larger to be played with hands, beaters, mallets, or sticks. Figure 30 
shows an example large-scale arrangement of two impact sensors and/or touch pads for ex- 
ecuting one-handed drum-rolls and deriving large amounts of control information. The fig- 
ure illustrates a larger-scale arrangement of two impact sensors and/or touch pads 3000a, 
3000b supported in the method's configuration by, for example, supporting beams 3003, 
3004 connecting to a common suspending clamp 3005 on an instrument-stand beam 3006 
on one side and joints 3007, 3008 to the sensors and/or pads 3000a, 3000b on the other side; 
though, clearly, other mounting arrangements are possible. The sensors and/or pads 3000a, 
3000b are separated from one another by a distance that permits a beater 3010, mallet, or 
stick to be held in one hand at the far end 3011 and rapidly rocked back and forth between 
the two sensors and/or pads so that the beater head 3012 impacts the sensors and/or pads. 
The beater may also be held at its center of mass or geometry 3013 and vibrated so that both 
the beater head 3012 and end tip 3014 of the far end impacts the sensors and/or pads; in this 
technique the player may orient the beater motion so as to simultaneously impact one im- 
pact sensor and/or pad with the beater head 3012 and impact the other impact sensor and/or 
pad with the beater end tip. In this playing technique it is advantageous to have provided 
for some regional differentiation of the impact sensors and/or pads; null/contact pads, for 
example can do this. Another arrangement is that of two impact sensors, one for the center 
area 3001 of an impact pad 3000 and the other for the outer rim area 3002, With the ability 
to differentiate regions of impact, and even non-impact applied contact regions and pres- 
sure, the portion contacted by the end tip and head can be differentiated. Further enhance- 
ment can be obtained by using a beater endowed with sensors; these can provide contact 
localization information, as well as hand grip information, which may be used independent- 
ly or in correlation with the information generated by the pads 3000a, 3000b. The resulting 
arrangement allows a performer with one hand to do a wide range of percussion and other 
control actions, leaving the other hand free for playing another instrument entity or express- 
ing visual gestures during performance. 

In a smaller scale implementation, one or more fingers can be used in place of a beater. This 
arrangement can be treated as an instrument element in itself to be used as part of other in- 
strument entities. 

Regardless of scale, it is noted that two such arrangements can be colinearly co-located but 
in 90-degree rotational offset. This creates a rectangular cavity for beats, fingers, etc. to be 
inserted and vibrated, and additional degrees of control. This can be generalized into arbi- 
trary polygonal cross-sections (triangles, pentagons, hexagons, etc.). 
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Video hand position and gesture 

A camera with appropriate real-time image processing may be used simultaneously or mu- 
tually exclusively as an instrument element as well as a video feed source for recording or 
performance. As such the camera may be treated as an instrument element mounted on an 
instrument entity, but can also be used as a self-contained instrument entity. For example, 
a camera could be aimed upwards and surrounded by illuminating lights. A performer can 
activate and control this instrument entity by putting a hand over the camera and executing 
various positions and gestures recognized by the image processing capabilities. 

Video Stage Tracker 

A camera may also be used to transform visual information observed from a stage into con- 
trol signals. The relevant image processing and recognition capabilities may advantageous- 
ly include identifying and tracking performer location and motions. 

Example Adapted Instruments 

This section discusses example manners and methods the invention provides by which a 
number of traditional vibrating element instruments can be enhanced by incorporating var- 
ious synergistic combinations of traditional components and the invention's instrument el- 
ements. 

Autoharp 

A traditional autoharp incorporates a plurality of strings, tuned to selected notes in a chro- 
matic scale, which are selectively damped by mechanical damping bars with cut-outs in the 
damping material that allow only selected strings to sound. A player selects and activates a 
damper bar associated with a chord and strums a portion or all of the strings, and only the 
undamped strings, namely those associated with the voicing of the chord, sound. Although 
at times considered a lower folk or beginning instructional instrument, the basic arrange- 
ment of the autoharp can give rise to a powerfully flexible instrument. 

In its simplest provision, the invention provides for an autoharp to be supplemented with 
sliders, switches and buttons for issuing control signals. In particular, a select group of but- 
tons or contacts can be operated by, or in conjunction with, the mechanical damper bars. 
This group of buttons or contacts may be used to control at least one of the following: issued 
note control signals for sound, lighting, and/or special effects, note assignments to one or 
more strum-pads, and/or the amplification of individual strings. The individual strings of 
the autoharp may have one or more of the following: a common pickup for the entire group 
of strings, a plurality of smaller pickups associated with sub-groups of strings, or a full plu- 
rality of individual pickups for each string. The pickups may be any of electromagnetic, pi- 
ezo, optical, etc. in their operation. In cases where a plurality of pickups is employed, 
signals from groups of strings or individual strings may be handled by multi-channel signal 
processing as described later (for example, treating the strings with differing degrees of 
equalization, chorus, reverb, pitch shift, dynamic filter sweeps, etc., and/or providing sep- 
arate noise gates, compression, limiting, amplitude control, etc.). In cases where each string 
has its own pickup, the plucking of a particular string may further be used to trigger a syn- 
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thesizer note, lighting, or special effect event, potentially using the amplitude of the pluck 
to set note velocity and potentially tracking the ongoing string amplitude and even harmon- 
ic structure variations as provided for in the invention and described later. Strum-pads may 
be provide for use in conjunction with strumming the strings or in conjunction with oper- 
5 ating the mechanical chord dampers. Controls may be provided for stored program recall 

of control signal assignments, strum-pad voicings, etc. as well as operational features such 
muting or sustaining of strum-pad notes, whether notes issued at the pressing of a chord 
damper bar are released when the damper bar is released or instead only when a new bar is 
activated, etc. These control features may also be controlled remotely, for example, with a 
10 foot controller, and/or implemented remotely in a separate signal routing, processing, and 

synthesis entity 120. 

Figure 3 1 shows an example of an enhanced autoharp implementation as provided for in 
the invention. A conventional autoharp 3100 with the usual arrangement of strings 3101 
and damper-bar activating chord buttons 3102 may be fitted with a long strum-pad 3103 
adjacent to the traditional strumming area, one or more shorter strum-pads 3104a, 3104b 
near the chord button area, a plurality of slider controls 3105 and control switches 3106, 

-= and control buttons for stored program recall 3107a, operational mode control 3107b, or 

:r : other features 31 0 7c. 

rg) As another part to the invention, the mechanical chord damper bar arrangement may be ad- 

vantageously replaced with a 12-note keyboard or similar arrangement for selecting which 
iU chromatic notes are allowed to sound. String damping control may be done mechanically 

^ although this requires damper bars to normally damp selected strings and let those wanted 

^ strings sound only when a key or button is depressed (rather than damping only unwanted 

25 strings when a key or button is depressed). In this way more arbitrary chords can be select- 

s' ed, chords can be dynamically changed at a resolution down to one pitch at a time, etc. Al- 

{!] tentatively, if a separate pickup can be provided for each string, mechanical string sounding 

\1 control may be replaced with electronic amplitude control. In the simplest form, all strings 

\U of various octaves of the same note are gated on and off by the depression of the key on the 

i|0 keyboard associated with that note. If the key depression-depth or total pressure on the key 

H is used as a volume control, the relative volume of all octaves of each pitch can be con- 

trolled independently from that of other pitches. If the key further has two-dimensional 
touch sensing, as with a null/contact touch-pad on each key, balance between various - typ- 
ically four - octaves can be continuously varied (for example left/right controls the balance 
35 between octaves 1 and 2 and in/out controls the balance between octaves 3 and 4, thus al- 

lowing arbitrary balance choices of the four octaves). The multi-parameter key control of 
the amplitude and mix of each sounded note is of particular value while the string sounds 
after the note is initiated. The keyboard, multi-parameter or not, can also be used to control 
similar aspects of note assignments and amplitudes of synthesizer notes initiated with each 
40 strum-pad. 

Figure 32 shows how the autoharp arrangement of Figure 30 can be adjusted to replace the 
chord button array 3102 and associated strum-pads 3104a, 3104b with a keyboard 3202 and 
one or more strum-pads 3204 positioned over the keyboard. 

45 
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Harps, Koras, Zithers, Kotos, Mbiras 

The enhancements of Harps, African Koras, Zithers, Japanese Kotos, African Mbiras, and 
other related instruments with a large array of hand-plucked vibrating elements are also 
provided for as part of the invention. As with the above autoharps, pickups may be used for 
all vibrating elements, or, advantageously, sub-groups of elements, or - most advantageous- 
ly - separately for each vibrating element. The pickups may be any of electromagnetic, pi- 
ezo, optical, etc. in their operation. The invention also provides for the instrument to be 
supplemented with strum-pads, touch-pads, sliders, switches and buttons for issuing con- 
trol signals and affecting internal operation and note-event handling modes. 

In cases where a plurality of pickups are employed, signals from groups of vibrating ele- 
ments or individual vibrating elements may be handled by multi-channel signal processing 
as described later (for example, treating the strings with differing degrees of equalization, 
chorus, reverb, pitch shift, dynamic filter sweeps, etc., and/or providing separate noise 
gates, compression, limiting, amplitude control, etc.). In cases where each vibrating ele- 
ment has its own pickup, the plucking of a particular vibrating element may further be used 
to trigger a synthesizer note, lighting, or special effect event, potentially using the ampli- 
tude of the pluck to set note velocity and potentially tracking the ongoing string amplitude 
and even harmonic structure variations as provided for in the invention and described later. 
Strum-pads may be provided for use in conjunction with plucking the vibrating elements. 

Harps, Koras, Zithers, Mbiras, and other related instruments with a large array of hand- 
plucked vibrating elements often have only selected pitches available; accidentals and ex- 
treme octaves typically are not represented. Many of these instruments allow for acciden- 
tals during playing, for example harp tuning levers and Koto string bends, while others, 
such as the Mbira, do not; in almost all cases extremal octaves are not supported at all (aside 
from execution of fundamental-muting string "harmonic chiming" to attain high octave 
pitches). With each vibrating element (or, less flexibly, groups of vibrating elements) pro- 
vided a separate pickup and audio channel, pitch shifting can be used to electronically ob- 
tain pitches not provided for by the natural form of the instrument as well as large 
expressive pitch bends that may also not otherwise be possible. 

Figure 33 shows an example Koto implementation provided for in accordance with the in- 
vention. In general a Koto includes a number of strings 3301 (typically 13 to 22 in number), 
a sounding bridge 3302, and a movable truss bridge 3303 for each string. The Koto 3300 
shown in Figure 33 is of the Vietnamese variety, traditionally strung with sixteen metal 
strings; in general any traditional Koto (or Chinese Sheng) can be adapted as will be de- 
scribed and may have any traditional number of silk, nylon, metal, or other material strings. 
In this example the Koto has been fitted with geared string tuners split into two groups 
3304a, 3304b to facilitate radical tuning changes and string replacement; other nominal 
string tuning mechanisms, including traditional friction pegs or slip knot systems, may also 
be used. Each string can be given its own pickup 3306.1 - 3306.n, at the bridge 3302; alter- 
natively, or in addition (should the string be such that electromagnetic, optical, or other 
non-piezo pickups be applicable) at a different string location in one or more pickup hous- 
ings 3305. Multi-channel signal handling as described earlier and later on can be used. The 
Koto can be fitted with a strum-pad 3310 and may also be provided with various additional 
sensors and controls as described earlier. Because of the unique pitch-bending arrangement 
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of the Koto involving varying the tension of the string on the non-plucked side of the mov- 
able truss bridges 3303, it may also be advantageous to provide strain gauges on far bridge 
3307 or via, for example, flexible attaching electrical cables on the truss bridges 3303 them- 
selves. 

Figure 34 shows an example Mbira implementation provided for in accordance with the in- 
vention. Here a traditional Mbira 3400 with three sets of tynes 3401, 3402, 3403 secured 
by a bridging pressure bar 3404 is shown. Here the traditional bar bridge may be replaced 
with individually adjustable bridge elements 3406.1 - 3406.n which may include separate 
piezo pickups for each tyne; alternatively electromagnetic or optical pickups may be pro- 
vided under each tyne. The Mbira may also be provided with one or more strum-pads 340 7, 
3408 which may be full length or partitioned into right and left segments 3407a, 3407b, 
3408a, 3408b for use with the thumbs of both hands. 

Any of these instruments may also be provided with vibrating element excitation employ- 
ing the methods presented earlier in association with Figure 24. In addition, as the timbre 
of these instruments is typically shaped by the sympathetic vibration of unplucked strings 
on the instrument, it may also be advantageous to excite, as a group or individually, a num- 
ber of vibrating elements such as tynes or strings using the methods described earlier; these 
sympathetically vibrating electronic elements, which need not be mounted to the instru- 
ment, can then produce audio signals that can be used for ambient effects. Alternatively, a 
dedicated group of sympathetically vibrating elements can be mounted to the instrument 
and excited mechanically rather than electrically. If the sympathetic vibrating elements 
have individual pickups, some of the elements can be selectively turned off or attenuated 
so as to thin out or spectrally sculpt the ambient effect. 

Single-course Guitars and variations 

One of the most versatile instruments available for the range of timbre and expression is the 
electric guitar which is sadly not often used seriously in music composition due to its ori- 
gins and significant role in popular music. (In fact, at this writing, even toy pianos are taken 
more seriously than the electric guitar!) Part of the reason for the immense range of timbre 
and expression is the fact that it is one of the few instruments where both hands can be in 
direct contact with the string. Another important reason is the range of timbres that can re- 
sult from string pickups followed by a wide degree of signal processing methods that have 
been developed and can be applied. Although there continue to be developments in basic 
electric guitar themes, the invention provides for significant enhancements of the electric 
guitar as a powerful instrument entity. 

An important first step is the provision of separate audio signal pickups for each string; 
these may be electromagnetic, piezo, optical, etc. This allows for multi-channel signal pro- 
cessing as will be described later (for example pitch shifting particular strings for big bass 
notes, enhanced processing for strings playing solo lines to stand out from strings playing 
background material, etc.). Strings may be given one or more dedicated or shared pickups 
at different points along the string's length so as to capitalize on the different harmonic 
structure and dynamics offered by different pickup locations. A plurality of pickups dedi- 
cated to the same string or same group of strings can be selected or mixed, potentially in 
adjustable phase relationships, statically and/or varying in time, on the instrument and/or 



LUDW 001/01 US 



externally. Further, selected strings may be excited by electromagnetic, piezo, or other 
methods to give a continuously sounding bowed effect whose inter-note attack can be con- 
trolled by various fretting techniques. Additional strings arranged to serve as a harp ele- 
ment, bass notes as on an arch-lute, or for sympathetic vibration may also be provided, as 
may tynes or other vibrating elements used in similar ways. Strum-pads, sensors, sliders, 
joysticks, buttons, touch-pads, actuators, etc. may also be added to issue control signals to 
any of signal processing, lighting, synthesizer, or special effects. Similarly, video cameras 
can be used to generate control signals and/or for video image feeds in performance or re- 
cording. 

Figure 35 shows an example electric guitar implementation in accordance with the inven- 
tion based on a Gibson model ES-335 guitar or other instruments of that style. The inven- 
tion's enhancements shown can be added on as modules, added collectively, or built-in. 
The core guitar 3500 features six strings 3501 J - 3501,6, a bridge 3502 which may provide 
separate piezo pickups for each string, an adjacent hexaphonic pickup module 3503 which 
provides separate electromagnetic, optical, or other pickups for each string, and additional 
individual string pickup module 3404 for the three lower pitched strings 350L4 - 3501.6, 
two shared pickups of a two-coil humbucking design near the neck 3505a and near the 
bridge 3505b, shared pickup volume and tone controls 3506, and a shared pickup selector 
switch 3507. Individual strings may be electronically excited by means of any of a hexa- 
phonic electromagnetic drive unit 3508, a hexaphonic pickup used as a drive unit (or de- 
signed as a drive unit) built into the forward shared pickup 3505a, or by mechanical 
excitation via the piezo elements in the bridge 3502. Individual string excitation, or even 
string activity in general, can be visually indicated by an LED (preferably high-brightness) 
array 3509 under the strings. Group string excitation may be realized by whichever of pick- 
up 3505a or 3505b is not in use, an additional module under the stings, or by group me- 
chanical excitation via the piezo elements in the bridge 3502. Individual string pickup gain 
normalizing adjustments for each hexaphonic pickup can be made available for screwdriver 
adjustment 3511a, 3511b on an add-on box or built-in panel 3512, shown in the Figure as 
an add-on box with generalized interface connectors 3519 on its back downward side. The 
instrument may also be provided with an array of control switches 3513 and sliders 3514, 
individual string pickup selector switches 351 6, sl video camera 351 7 aimed at the playing 
area but also useful for hand posture and gesture control, and an area 351 8 for additional 
controls, touch-pads, string or tyne arrays, etc. 

Figure 5 shows an example electric guitar implementation in accordance with the invention 
based on a Gibson Explorer model guitar; the invention's enhancements shown can be add- 
ed on as modules, added collectively, or built-in. The core guitar 500 includes six strings 
501.1 - 501.6, a locking-nut vibrato bridge 502a with string-tension "whammy bar" 502b 
and fine tuners 502c, a hexaphonic electromagnetic or optical pickup module 503, a shared 
pickup 505a near the neck and another 505b near the bridge, a shared pickup volume con- 
trol 506, shared pickup selection, mixing, phase, and coil-shunt switches 507. The instru- 
ment may also be fitted with a hexaphonic electromagnetic excitation module 508 with 
string drive indicated LEDs 509. Also provided for by the invention are controls for creat- 
ing control signals: an array of switches 513, knobs 514a, one or more expression wheels 
514b, one or more joysticks 514c, two side-by-side strum-pads 516a, 516b which can be 
operated as one long strum-pad, chord buttons 520 as described for the autoharp but here 
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used without strings and rather in conjunction with the strum-pads, a touch-pad 522, two 
miniature keyboards 521a, 521b, a miniature harp/sympathetic string set 524 with group or 
individual pickups partitioned in string triads with separate bridges 523a, 523b, 523c and 
tuning heads 525, and a plurality of impact sensors. Internally the whammy bar may operate 
a discrete or continuous position sensor and the instrument may also include motion sensors 
(such as accelerometers) or position detectors (radio, ultrasonic, optical, etc.). The general- 
ized interface connector area 519 in this instance is shown built into the instrument. 

Baroque and 12-string Guitars, Lutes, Tars, Setars, Saz, Oud, 
Mandolins, Mandolas 

These instruments involve double-strings. In addition to the techniques and additional in- 
strument elements, each double-string pair may share an individual pickup, or each string 
within in a double string pair may have its own pickup. At this writing the best more for the 
latter appears to be' piezo pickups at the bridge due to limitations in localizing magnetic 
fields for such close geometries but optical or other methods could be devised. With a sep- 
arate signal for each string within in a double string pair, either of the strings can be selec- 
tively disable, pitch-shifted, equalized, etc, along with other capabilities such as adjustable 
balance, stereo spatial output, opposing location modulation trajectories, etc. Further, as a 
combined double-string signal would confuse audio-to-note information conversions, sep- 
aration of the string signals for a given string pair enables control extraction such as con- 
version to MIDI note functions. 

Figure 36 shows an example of an adapted European arch-lute with a mix of single strings 
and double string pairs. The European arch-lute 3600 with various ranges of extended fret- 
ted and unfretted bass strings is a noble instrument which with amplification, signal pro- 
cessing, and multi-channel signal handling can contribute greatly to electronic music. 
Figure 36 shows a close-up of the strings at the bridge 3620 for a fourteen string version of 
the lute with six extended-length unfretted bass strings: the two highest-pitched melody 
strings 3601, 3602 and the six extended-length unfretted bass strings 3609 - 3614 are single 
while the remaining courses 3603-3608 are double-strung. Of the double-strung strings the 
higher pitched ones are typically in unions while the lower pitched ones are paired in oc- 
taves. It is understood than many other string configurations, varying in how many strings 
and which are single or doubled, exist and can be adapted as described herein. In fact no 
electric lutes are known at this time, thus it is novel simply to include a group piezo pickup 
in the bridge 3620 for all the strings so as to bring the instrument into the world of ampli- 
fication and signal processing. A next level refinement would be to provide separate group 
pickups for the extended bass strings and the rest of the strings so that special equalization 
or other effects can be applied to the bass notes in a manner differing from the other strings. 
A next level of refinement would be for the bridge 3620 to provide an individual pickup for 
each individual single string course or string-pair, while a final refinement provides a sep- 
arate piezo pickup for every individual string on the instrument. 

It is understood that various controls, strum-pads, etc. may also be added in the manner de- 
scribed for previous instrument examples. It is also understood that the methods described 
also apply to other double-strung instruments such as 12-string guitars, Saz, Oud, Mando- 
lin, etc. Many of these instruments may also benefit from an additional set of unfretted bass 
strings as incorporated in the traditional European arch-lute. 
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Pedal Steel Guitars 

The pedal steel guitar is a remarkable instrument in that the pitches of individual strings are 
changed as a group by a hand-held metal slide and relatively within the group by mechan- 
ical bridge arrangement, usually called a "changer," which changes the tension on one or 
more selected strings in response to the action of a given foot-pedal or knee lever. The basic 
sound of the steel guitar is very attractive and it is possible to tastefully play Bach chorales 
and hymns on the instrument. Years of incremental development have lead to specific stan- 
dard pedal and knee lever configurations that are widely accepted. Variations are some- 
times difficult to implement because of mechanical limitations to provided adjustments. 
Because of the commitment involved in mechanically establishing an alternate pedal and 
lever configuration, immense experience and/or a computer-aided design tool may be re- 
quired to make valuable accomplishments. By providing a separate pickup for each string, 
retuning can be done electronically, supplementing or replacing the traditional mechanical 
mechanisms. As with other adaptations of instruments described thus far, each string can 
also be processed separately or in groups as desired, allowing for mixes of timbres, and au- 
dio-to-control signal extractions can be used to control synthesizers, signal processing, 
lighting, and special effects. Further, the nearly fixed position of the picking hand and the 
freedom of some fingers in adapted playing techniques allow usage of miniature keyboards 
and strum -pads in the picking area as will as use of the wrist to control parameters. Infor- 
mation from the mechanical or electronics pedals and levers and the steel bar position can 
be used to control the pitches assigned to a strum-pad. The bar itself can have a control area 
built-in, detecting applied pressure, for example. 

Figure 37 shows an example pedal steel guitar adaptation as provided for by the invention. 
A traditional pedal steel instrument arrangement 3700 is used here. The bar 3702 may or 
may not be provided with internal sensors or controls. The position of the bar 3702 over the 
instrument 3700 may be sensed by various means (changes in round-trip string resistance, 
capacitive sensing, etc.) within the instrument 3700. The instrument may include a tradi- 
tional mechanical changer 3 740 with linkages to pedals 3741 and levers 3742 or may be 
fixed if all pitch bends are to be done electronically. With a mechanical changer, the pedals 
3 741 and levers 3 742 may be fitted with sensors that measure and convert their displace- 
ment into control signals. Other control signals may be issued by a wrist operated expres- 
sion wheel 3714a (which in practice works well for overall volume control), one or more 
pedals 3714b, some of which may be fitted with side-mounted spring-lever controls 3714c. 
The pedal rockers can be used for various control functions, but offer a way of bending 
pitches electronically and holding them without having to devote a foot or knee to that pur- 
pose. In contrast, the side-mount spring levers, as well as a spring loaded version of a rocker 
pedal, directly emulate the spring-return operation of a traditional pedal steel guitar's ped- 
als and levers. The pedals can also be used for the introduction or variation of signal pro- 
cessing on one or more selected strings and can be configured to make such effects while 
simultaneously changing pitch of selected string or strings. The adaptation also provides for 
a strum-pad 3716 and miniature keyboard 3721 in the picking area for use by idle fingers. 
Finally, a foot-controller foot-switch unit 3742 for use in selecting stored programs, con- 
trolling signal processing, issuing notes or chords, operating drum machines, etc. is also 
provided. 
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Sitars 

The Sitar is an extraordinary rich instrument that is well-suited for the particular structural 
details of classical indian music. It includes a number of drone strings, only one or two of 
which can be fretted in any musical way, a single melody string, and an octave pair of un- 
fretted high pitch strings, called the "chikori" (Western spellings vary) used for a variety of 
purposes including quite effective rhythmic accents, all sharing a common sloping bridge 
that cause the aforementioned strings to twang to a degree determined by the slope of the 
bridge. A set of sympathetic strings with their own sloping bridge, which in some tech- 
niques can be arpeggiated and/or used as a small harp to a limited extent, is also provided. 
The Sitar features a selected combination of both brass and steel string types which have 
important essential distinctions in timbres 

Uses of the Sitar in Western music tend to fall into two categories: one where only the mel- 
ody string, along with any sympathetic string action, is used, and another where the sitar' s 
many drone strings force the tonality into the standard Indian tonal development system 
(rich and extraordinarily beautiful as it is). A Sitar-like sloping bridge has been successfully 
put on a guitar (the Jerry Jones Coral Sitar" heard in many Motown-era popular recorded 
songs), but all that remains is the twang as the genius of the Sitar holistically has been omit- 
ted. 

The invention provides for a powerfully rich adaptation of the Sitar by combining the tech- 
niques described thus far with the signal routing, processing, and synthesis techniques to 
be described later and, as with the previous examples, inherent aspects of the instrument. 

Figure 4 shows an adapted sitar as provided for in the invention. The core instrument is a 
standard Indian Sitar 400 with a standard melody string 401.2, any one of a number of pos- 
sible stringings of drone strings — here two 401.3, 401.4, are used, and the chikori pair 
401.5a, 40L5b all sharing the common sloped bridge 402a. Also part of the core instru- 
ment, but not showed explicitly in the Figure for the sake of clarity, are the sympathetic 
strings, typically eleven in number, with their own sloping bridge 402b and multi-length 
termination area 402 on the neck under the Sitar' s curved frets. Because of the curved frets 
and the flat bridge 402a, the drone strings 401.3, 401.4 will not sound accurately in pitch 
for nearly all the frets; further, the characteristic extensive bending of the string, often the 
interval of a major fourth or more, requires quite a bit of area on the sitar neck (namely the 
entire bottom half of the neck) and as a result the drone strings' use in melody is essentially 
nil. However, it is possible to add an additional melody string 401.1 with its own lower 
bridge; this string would be pushed up to get the bending effects that the original melody 
string 401.2 is pulled down for. Both melody strings are provided with fine tuning adjust- 
ments such as the typical beads 430\ these may also be used on drone strings and/or alter- 
nate tuning mechanics may be substituted throughout the instrument. Also provided, 
though not essential, is an additional set of strings 424 that can be used as a harp in addition 
to the limited-access aforementioned sympathetic strings. In the Figure these are shown 
well-removed from the neck at the top of the instrument, but the assembly 424 could be 
brought forward and put in the position currently occupied by the keyboard 421. The key- 
board 421 is also not essential but is easy to support electronically, employing the method 
associated with Figure 8, in conjunction with the strum-pad 416a located on the plucking 
area of the neck and the strum-pad 416b located on the thumb-rest area of the neck; further, 
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both the keyboard Harmonium and keyboard synthesizer are popular in Indian classical, 
modal, and Ghozal folk music. The strum-pads are provided because of there particular po- 
tential use in the Rag (Raag, Raga) form both in laying out the scale of the Rag and melody 
expositional fragments which are repeatedly drawn upon. 

Important to the adaptation is the pickup assembly 403 which provides a separate pickup 
for each melody string, each drone string, and either the chikori pair or its individual 
strings. The separate outputs allow for pitch shifting of individual strings; in particular, the 
pitch shifted retunings of the drone strings and chikori can be made while playing. If the 
pickup is electromagnetic, the brass strings cannot be used. There is the opportunity here 
for alternative stringing systems, particularly if pitch shifting of individual strings is used 
to create larger pitch- shifts, but the character of the brass strings is beautiful and can be cap- 
tured. One method is to use an optical pickup for the pickup assembly 403. Another more 
radical approach is to replace the sloping bridge 402 with a standard bridge arrangement 
fitted with individual piezo pickups and to create the twanging using the off-bridge sitar 
plate discussed in association with Figure 21. 

The additional melody string can be tuned in union or in an interval to the original melody 

n string; because the have separate audio channels they can be processed differently or be lo- 

=0 cated at different positions in the stereo sound field. Further, the additional melody string, 

29 strum-pads, and addition string assembly serve to expand an important orchestrational as- 

1= pect of seasoned Sitar technique, namely a constant variety of timbres and effects with at- 

*M tention constantly shifting among them. Finally, the electronic pitch shift retuning 

:£ capabilities allow for hitherto impossible tonality shifts within the Sitar environment, while 

]i : the electronic pitch shift pitch-bend capabilities allow the drone strings to obtain pitch 

25 bending and the melody strings to be harmonized in a pitch-modulated manner. 

O It is also possible to carry simplified versions of the Sitar tonality into more Western instru- 

ment formats. Figure 38 shows an example flat-necked instrument 3800 with a five string 
l *2 section emulating a sitar string arrangement and several additional strings used for bass or 

other accompaniment. The sitar emulation section involves strings 3801.1 - 3801.5 sharing 
a common bridge 3802a providing a separate piezo pickup for each of the strings. The first 
' four of these strings terminate their vibrations at the nut 3802c and are tuned with neck tun- 

ers 3851, while string 3801.5 acts as a one-string fretted chikori and terminates its vibration 
at one of the mid-neck frets. The best terminating fret may depend on the chosen tuning and 
thus terminating holes for this string might be provided at several different frets, for exam- 
^ pie from the 4th fret to the 12th. String 3801.5 requires a different tuning arrangement and 

in order to accommodate a selection of termination points the tuner 3802 may be located 
on the body, perhaps with the tuning head in a protective cut-out area of the body. Another 
important item is that string 3801,4 is brass while the others are steel, creating the Sitar so- 
nority. The additional strings, shown in Figure 38 as six in number 3801.6 - 3801.1, can 
serve as bass strings or harp strings and may be terminated on a rapid return bridge (such 
as the Hipshot Trilogy product) to facilitate flexible use during performance, particularly 
because of the width of the fretted neck; alternatively these strings may be unfretted as on 
the European arch-lute. These strings are most likely best served by steel strings and thus 
each string may be given its own pickup either at the rapidly retunable bridge assembly 
^ 3802b or with a body-mounted multi-channel pickup 3805. 
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Finally, in lieu of a sloped bridge or the arrangement of Figure 8, it is also possible to create 
a synthetic and/or enhance the actual sympathetic/buzz/twang aspect of these instruments 
with signal processing techniques. A precisely-set, short audio delay (for example 3 msec, 
6 msec, or 12 msec for various octaves of the note E in A440 tuning) with high positive 
feedback acts as a resonator that easily distorts when excited by an audio input at its reso- 
nant frequency. This resolutes in a swelling and diminishing twanging similar to that cre- 
ated by the sloping bridge. This sort of effect can be found in one of the example presets of 
the Boss SE-70 stereo signal processor. The effect can be considerably enhanced by follow- 
ing this resonator with a low-speed sweeping flanger and a low- speed sweeping chorus, 
each with their own moderate amount of positive feedback, and made more emotionally 
powerful by low-speed auto-panning location modulation. Figure 39 shows an example 
multiple-pitch sympathetic/buzz/twang resonator feeding an input signal 3900 into banks 
of short audio delays 390 LI - 3901. n with high resonances tuned to each selected pitch, 
each followed by a dedicated low-speed sweeping flanger with moderate resonance 3902.1 
- 3902.n, a dedicated low- speed sweeping flanger with moderate resonance 3903.1 - 
3903. n, and a low-speed auto-panner 3904 A - 3904.n. All autopanner outputs are summed 
by a stereo mixer 3905a, 3905b to create a stereo output 3906a, 3906b. It is important that 
all the sweep oscillators be slightly detuned to minimize repetitive discernible patterns. The 
number of individual resonant pitches supported would involve every note in the scale and 
perhaps an octave of some selected notes such as the tonic and fifth; this could range from 
5 to 16 pitches. It is understood that variations, simplifications, and elaborations of this ex- 
ample arrangement are possible. 

Pipas 

Like the Indian Sitar, the Chinese Pipa features a mix of string types, here involving steel, 
silk, and composites of these. The Pipa (and to some extent its Japanese colleague, the Bi- 
wa) also has a rich ancient tradition yet contemporary appeal. Despite being far less known, 
it is capable of a great range of sonic techniques, with a high number of formal playing tech- 
niques as compared to many other instruments. Included in the extensive technique suite 
are a number of body taps and impacts made on the large front surface of the instrument. 

As with the above example adaptations, the invention provides for adaptations of the Pipa 
that involve instrument elements of the invention set to capture and complement the char- 
acteristics of this rich and deep instrument. Again, piezo bridge pickups are felt to be the 
best mode for capturing the subtle acoustic nuances of the different string types, and a sep- 
arate pickup for each string permits the usual multi-channel signal processing possibilities 
and control signal extraction for controlling synthesizers, signal processing, lighting, and 
special effects. Body taps and impacts can be directed towards impact sensors, and the usu- 
al possible collection of extra strings, keyboards, strum-pads, touch-pads, sliders, switches, 
buttons, sensors, etc. may be added to the large open area for instrument augmentations. In 
particular, strum-pads and a bank of harp strings are especially applicable due to the com- 
mon use of pentatonic scale sweeps and repeated short melodic figures during develop- 
ment. Also especially useful for incorporation into Western sonic structures would be the 
addition of a bank of bass strings and the use of signal processing as the Pipa tonal range, 
though fascinating, arrives somewhat unfocused on undeveloped Western ears unfamiliar 
with the instrument repertory. 
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Figure 40 shows an example adapted Chinese Pipa as provided for by the invention featur- 
ing various impact sensor arrays 4003a, 4003b, 4003c, a keyboard 4004, slider array 4007, 
switch array 4008, touch-pad 4009, strum-pad 4010, and separate piezo pickups for each 
string 4001.1 - 400L4 at the bridge 4002. 

Figure 41 shows another example adapted Chinese Pipa as provided for by the invention 
featuring impact sensors arrays 4003a, 4003c, strum-pad 4010, an unfretted harp string ar- 
ray, involving a terminating body tuning bridge 4110a and body bridge 4110b, and an un- 
fretted bass string array involving body bridge 4111a and head tuning bridge 4111b with 
tuners 4113. The two string arrays may have individual string or group pickups located ei- 
ther in the body bridges 4111a, 4111b and/or on the body at an interior portion of the string 
vibration 4114, 4115. 

It is understood that many other combinations of instrument elements are possible. 

Erhus, Dilruba, Esraj, Sarangi, Kamamcheh 

Each of these bowed instruments has its own rich tradition and special tonal qualities. Many 
of these instruments are used to accompany vocals or even to replace a singer due to the 
vocal quality of the instrument. 

The invention provides for adaptation of these instruments involving instrument elements 
of the invention set to capture and complement the characteristics of the traditional instru- 
ment and its musical traditions. In particular, in addition to the vocal quality of the sounds, 
bowing is a more conspicuous part of the sound as opposed to Western bowed instruments 
which encourage burying the perception of bowing logistics in favor of overall smoother 
tones. 

Again separate pickups may be used for each string: electromagnetic, piezo, and/or optical 
as appropriate for the type of string material, mounting arrangements, and other engineer- 
ing considerations. A separate pickup for each string permits the usual multi-channel signal 
processing possibilities and control signal extraction for controlling synthesizers, signal 
processing, lighting, and special effects. Those instruments with sympathetic strings, such 
as the Esraj, Dilruba, and Sarangi, may also include pickups for those strings as described 
in previous example instrument adaptations. 

Because each string has its audio channel picked up intimately with the string, it is possible 
to diminish some effects of the body resonance and replace it with electronically created 
resonances. In particular, vocal sounds are known to appeal to the ear as vocal in nature due 
to the relative center frequencies of a pair of predominant resonances as illustrated in Figure 
17. Through electronic synthesis of these resonances the vocal character of the instrument 
can be changed and, in fact, varied over time if one would dare to make the vocal bowed 
instrument literally sing. 

Further, because of the somewhat different role of bowing, more attention can be paid to 
collecting control information from the bow. However, the bow sensor techniques de- 
scribed can also be used to great advantage in Western bowed instruments. Figure 42 shows 
a bow fitted with sensors to gather information from the hand, bow hairs, and bow motion. 
The bow 4200 fitted with bow hair 4201 as usual, may include a bow-hair tension sensor 
4202, a free finger pressure-sensor and/or null/contact touch-pad (or pressure-sensor array) 
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4203, a handle grip 4204 pressure-sensor 4205, and an internal accelerometer 4206 to de- 
termine bow direction and acceleration. These control parameters can be transmitted to the 
instrument by attached electrical or fibre cable, wireless optical, wireless radio, or other 
means. As a simple example, the free finger controls the choice of resonant vowel formants 
for the instrument while the hand grip pressure 4205 or pressure on the free finger control 
4203 may be used to control the sympathetic string amplitude or a signal processing param- 
eter. The bow tension and accelerometer measurements can be used to control emphasis 
signal processing or darkening lowpass filtering. 

Flutes and Recorders 

Reed instrument layouts have been used in wind controller products by Akia and Yahama. 
However, flute-like (embouchure air hole) and recorder- like (fingers normally down) in- 
struments have to date not be used as models or methods for electronic instrument control- 
lers. 

It is noted that some types of Western flutes have at least some open holes, many folk and 
non- Western flutes have only open holes, and some flutes and recorders have at least one 
hole that is open but is operated by a levered key. In the discussion below, the flute example 
is considered to be purely closed hole and key operated while the recorder example is con- 
sidered to be purely open hole without levered keys; the thus illustrated techniques can be 
freely applied to other hole and lever arrangements of a particular instrument variant. Fig- 
ure 43 shows adaptations of a Western (closed-hole) flute and an (open hole) recorder lay- 
outs with pressure-sensors, or small pressure-sensor array touch-pads, replacing sites of the 
keys, together with air turbulence measurements, and air pressure average measurements 
as provided for in the invention. In either instrument adaptation, the instrument may be 
maintained as a sounding instrument or used as a model for an all electric controller. In a 
sounding version, an attached or internal transducer may be used to provide both an audio 
signal for processing and a means for note event control extraction (for example, using 
pitch-to-MIDI technologies such as those of, or superseding, the Roland CP-40). 

In the example flute and example recorder shown in Figure 43, the body area around the 
flute embouchure wind opening 4301 and recorder wind opening 4351 may experience air 
pressure changes and turbulence which may be measured with sensors and signal process- 
ing as described earlier; if the sensors and wiring to them are mounted securely in low pro- 
file the instrument behavior and sound will not be noticeably affected. Alternatively, in 
non-sounding versions, more obtrusive internal sensing of air turbulence and/or air pres- 
sure may be employed. 

In a sounding adaptation of the closed hole flute 4300, the area of the keys which contact 
the fingers 4302 can be covered with simple switches, a pressure-sensor, or a pressure- sen- 
sor array. Alternatively, in a non-sounding controller adaptation of the closed hole flute 
4300, the area of the keys which contact the fingers 4302 can be replaced by simple switch- 
es, a pressure-sensor, or a pressure- sensor array. In the case of the open hole recorder, sim- 
ple switches, a pressure- sensor, or a pressure-sensor array can be put around the perimeter 
of any of each single-hole 4353, each double-hole 4354, and the thumb-hole 4355. Because 
of special playing techniques associated with the double-holes (i.e., "half-covering") and 
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thumb-hole (thumb tip flip or other "half-covering" methods), these areas may be handled 
with more specialized switch and/or sensor arrangements. 

For the most part such hole-positioned and key-positioned sensors may be used to assist in 
issuing note events but ranges of additional technique can be developed for more sophisti- 
cated control. A less technique-oriented approach would be to put simple switches, a pres- 
sure-sensor, or a pressure-sensor array in an area 4306, 4356 where a thumb is otherwise 
only used for supporting the instrument. 

As with the other instrument examples, it may also be advantageous to place additional in- 
1 o strument elements such as strum-pads, touch-pads, sliders, switches, buttons, other sensors, 

etc. on the body of the instrument. 

4.10 Gongs, Bells, Cymbals, Chime Bars, other Metallaphones, and 
Acoustic Drum Heads 

^ Gongs, bells, cymbals, chime bars, xylophones, and other metallophones, as well as the 

stretched heads of acoustic drums, can be problematic to amplify because they typically un- 
dergo significant displacement motion when struck yet their sound may alter significantly 
'•^ if this motion is restrained and/or if a surface transducer is attached to them. It is noted that 

£ there are many types of musically useful non-stereotypical gongs with widely varying tim- 

12 bres, including for example the non-crashing, pitched Indonesian gongs with close-set 

j?= overtones which beat at low frequencies creating a complex tremolo effect that sounds in 

]i: many of these instruments very similar to pitch vibrato. 

M The invention provides for quality audio signal capture from these types of instruments be- 

25 cause of their musical usefulness, the rich possibilities for signal processing their sounds, 

~ and the visual appeal of their playing in a performance situation. Figure 44 shows how an 

jTj optical pickup may be created for a suspended gong; this technique may also be used for 

u many other types of metallophones and acoustic drum heads. In the example, a gong 4400 

ip is supported by small ropes 4401a through holes 4401b or other means. Etched into, pol- 

rfo ished into, sprayed onto, adhered onto, etc. the gong 4400 is a reflective or refractive area 

4404 which may be unstructured or may include a reflective pattern or refractive properties. 
One or more, typically two, light sources 4402a, 4402b, typically coherent and/or modulat- 
ed, direct light beams 4403a, 4403b to the reflective or refractive area 4404. The materials, 
light sources, and geometry are arranged so that the return light path 4405 is modulated in 
35 amplitude and/or intensity when the gong vibrates at audio frequencies. Reflective or re- 

fractive light 4405 from the gong is directed at one or more light detectors 4406 which use 
the resulting light amplitude or intensity modulation to create an audio signal. Lowpass fil- 
ters are used to remove subsonic signals resulting from the swaying of the gong. Figure 45 
shows an example metal bar which can be mounted so that extremes of impact displace- 
40 ment are relatively confined. In this arrangement the bar 4450 is suspended by small ropes 

4451a through holes 4451b or other means. The vibration of the bar may be captured in a 
number of ways; illustrated are a similar optical sensor 4452, and electromagnetic pickup 
4453 should the bar 4450 be ferromagnetic or have a ferromagnetic plating or attachment 
near the pickup, and a capacitive plate 4454 which can be used to measure the varying ca- 
45 pacitance between the plate 4454 and the bar 4450 via wires 4455a, 4455b connected to the 

plate 4454 and a conducting support rope 4451a or other means. Similar methods may be 

— 59 — 
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employed for acoustic drum heads. Not shown is the case where a bar or tyne is tightly se- 
cured on one end; in this case not only may the bar be amplified by similar methods but also 
by a piezo sensor in the support as in the case of the Mbira discussed earlier. 

It is noted that these pickup strategies all pick up localized vibrations from the metallo- 
phone. As with instrument strings, the produced timbre will vary widely with the selected 
pickup area. It is therefore provided for in the invention that multiple pickup areas may be 
used, permitting multi-channel signal processing to be applied to a single gong in a way like 
that described earlier for instrument strings. 

Figure 45 shows example gong arrays as part of a one-hand or two-hand percussion instru- 
ment stand 4500. The top tier of gongs 4 50 LI - 450Ln in this example are shapely gongs 
such as the Indonesian gongs described earlier. The middle tier of gongs 4 502 J - 4502,m 
in this example are flat gong plates such as those found in Chinese percussion. Also shown, 
in this example instrument are two sets of acoustic drums 4603 configured for one-hand 
rolls as described earlier for electronic pads; the vibration of the heads of these drums are 
captured as per the described methods for gongs and bars. 

Alternative Audio and Control Signal Sources 

Historically new instruments have been created through incorporation of not only newly 
developed technologies but also newly discovered phenomena. In this section recently 
available understanding of largely unrecognized or unutilized processes are adapted by the 
invention for use in generation audio and/or control signals. 

Chemical Oscillations, Patterns, Waves, and Rhythms 

The Belousov-Zhabotinskii reaction [Tyson] and many similar "non-equilibrium" chemical 
reactions exhibit oscillatory and animated pattern-forming wave propagation and mathe- 
matical chaos effects which can be visually and electrically monitored [Gray, Scott]. These 
behaviors are the result of nonlinear dynamics governing the evolving reactant concentra- 
tions varying within the mixture over time [Nicolis]. Varying types of electrodes can be 
used to measure component reactants independently. If multiple electrodes are used, differ- 
ing but correlated waveforms are produced simultaneously, useful for both control and spa- 
tial timbre formation methods described later on. To some extent these reaction processes 
may also be controlled [Ruoff; Nagy-Ungvarai et. aL] via electric fields, reactant modula- 
tion, etc.- means that in fact can be controlled directly or indirectly by electrical signals. 
Chemical indicators may be used to visually enhance the observable contrast of pattern an- 
imation [Tyson; Orban et. al.]. The resulting animated patterns, which range from swirling 
spirals to complex tidal forms - remnants of 1960's animated hallucinogenic iconography 
- can be captured by video camera. The character of the patterns have visual and intuitive 
appeal and familiarity because they readily occur in biology, geology, and other parts of 
nature [Nicolis, Baras]. Populations of these chemical systems can be coupled by various 
means and as thus are observed to have rhythmic and turbulent behaviors [Kuramoto]. 
These various dynamical properties of non-equilibrium chemical reactions can be adapted 
to create a new exciting class of instrument entities and performing environments which are 
described herein. 
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5.1.1 Chemical Oscillators as Sound S urc s 

In their simplest form, these chemical reactions act largely as simple oscillators [Tyson; 
Gray, Scott]. The oscillations are the result of nonlinear dynamics governing the evolving 
reactant concentrations varying within the mixture over time and typically are in the form 
of slowly evolving limit cycles [Field, Noyes; Gray, Scott]. Each reactant-monitoring elec- 
trode then produces an oscillatory signal for the duration of the oscillatory concentration 
variation of that reactant. 

In practice most oscillations occur at very slow rates, for example with a period of 40-60 
10 seconds, and have a short life time, for example under a hundred cycles, unless reactants 

are refreshed. The design of wider ranges of chemical oscillators has been investigated [Ep- 
stein; Epstein] and in that it is conjectured that chemical oscillations may drive insect wing 
vibrations it may be possible to design triggered chemical oscillators that oscillate at audio 
rates with various oscillatory durations. Such chemical reactions, when electrically moni- 
15 tored, can be used directly as sound sources in the same manner as an electromagnetically- 

monitored or piezo-mechanically-monitored guitar string. 

Less speculatively, recorded measurements of known slow short-lived oscillatory chemical 
O reactions [Gray, Scott] may be captured and processed as "audio samples" which can be 

;~ pitch-shifted and spliced for arbitrary duration with conventional audio sampling technol- 

jy ogy. Further, mathematical models of these oscillatory behaviors [Field, Noyes] can be nu- 

JTI merically simulated and altered so as to change rate, duration, and other attributes [Wang, 

% Nicolis] as per model-based audio synthesis. Such numerical models then add a new non- 

q acoustic class of modeled elements to the well establish acoustically vibrating ones such as 

p strings, pipes, tynes, membranes, etc., and as with the acoustic-based models, can be adapt- 

ed and extended to create yet other new effects. 

u\ 5.1.2 Chemical Patterns as a Dynamic Controller 

^ The inherent time scales of visual and electrically measurable pattern evolution in most of 

the well-know non-equilibrium chemical reactions, along with their potential for direct and 
indirect electrical controllability, makes these non-equilibrium chemical reactions interest- 
ing candidates for the generation of control signals. The invention provides for the spatial 
patterns of these non-equilibrium chemical reactions to be measured and converted into 
control signals and potentially, with any of several chemical processes, to control via con- 
trol signals and/or to video capture for display or recording. The invention provides that 
these measured control signals may be used to control any one or more of note events, sig- 
nal processing, lighting, or special effects. 

The invention provides for spatial patterns of these non-equilibrium chemical reactions to 
be measured electronically by specific types of electrodes [Gray, Scott] and/or via a video 

40 camera combined with image analysis, parameter extraction, and control signal assign- 

ment. If electrodes are used, these may be of various types, including those responsive to 
variations in specific families of ion concentrations [Gray, Scott] as well as those used to 
measure electric fields, potential differences, electrical resistance, etc. These electrodes 
may be distributed spatially in one, two, or the dimensions. Figure 46 illustrates spatial ar- 

45 rays of electrodes which may be used for measurement, as well as control, in two-dimen- 

sional and three-dimensional configurations. In an example two-dimensional 
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configuration, a shallow vessel 4610 has its bottom surface fitted with a two-dimensional 
array of electrodes 4611 selected from among the types cited earlier. These electrodes con- 
nect by wires 4612 to interface electronics 4613 that extracts the measurement information, 
from this extracts parameters (potentially under stored program control), and then assigns 
5 these parameters (potentially under stored program control) to outgoing control signals 

4614. Some of the electrodes 4611 may be measurement only, some may be control only, 
and some may be dual-use. Any control capabilities and/or stored program recall may be 
controlled by incoming control signals 4615. In an example three-dimensional configura- 
tion, a volume vessel 4620 has its bottom surface fitted with a three-dimensional array of 
10 electrodes 4621 selected from among the types cited earlier. These electrodes connect by 

wires 4622 to interface electronics 4623 that extracts the measurement information, from 
this extracts parameters (potentially under stored program control), and then assigns these 
parameters (potentially under stored program control) to outgoing control signals 4624, 
Some of the electrodes 4621 may be measurement only, some may be control only, and 
15 some may be dual-use. Any control capabilities and/or stored program recall may be con- 

trolled by incoming control signals 4625. Also illustrated in the three-dimensional example 
are an example reactant intake tube 4632 and example reactant outtake tube 4634 through 
p which new reactants 4631 may be introduced and old reactants 4633 may be extracted; this 

ip arrangement can be used to vary the relative concentrations of chemicals in the vessel or 

0:0 maintain balance through reactant refreshing. The exchange rates and relative concentra- 

1^ tions levels can be electrically controlled by conventional chemistry instrumentation means 

*M and thus the invention provides for these to be potentially controlled by incoming control 

;jr signals. It is understood that multiple intakes and outtakes such as 4632, 4634 may be uti- 

]Z lized, and that the intakes and outtakes may have interior spatial distribution structures 

'25 within the vessel 4620. It is also understood that these methods would also be applicable 

for one-dimensional or two-dimensional arrangements and other vessel shapes such as 
iTj dishes, tubes, etc. 

^ It is also possible to measure the evolving chemical patterns with a video camera, particu- 

:iL larly when differentiating visual indicator compounds [Tyson; Orban et. al.] are introduced 

into the mixture. Figure 47 shows an arrangement where evolving chemical patterns 4700 
in the dish 4610 of Figure 46 are illuminated with light sources 4701 and visually moni- 
tored by an overhead camera 4702; here the light sources and camera are supported by ar- 
bitrary stand methods 4703. The invention provides in particular for the video camera 4702 
to be combined with simple to complex real-time image processing operations such as po- 
sition threshold detection, edge detection location, pattern detection, pattern area measure- 
ment, pattern rate of change, etc. to derive a multitude of parameters which may be mapped 
into control signals. Because visual chemical indicators can be used to identify localized 
concentrations levels of specific chemicals, the aforementioned pattern recognition tech- 
^ niques may in some configurations conceptually be used in place of the potentially more 

costly chemical-measurement electrode arrays and supporting electronics. 

The invention provides for the aforementioned arrangements to be used as an interactive 
chemical performance environment. Outgoing control signals generated by the spatial 
chemical patterns may be used to control any one or more of: note events, timbre modula- 
45 tion, lighting, and special effects. Incoming control signals provided by or extracted from 

audio signals, electronic instrument elements, real-time sequencers, actuators, video cam- 

— 62 — 
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eras, or body position indicators (gestures, dance, stage position) can be used to control the 
evolution and influence the shapes of the chemical patterns. Video of the patterns may be 
displayed on monitors or projected, via video projector, onto the stage area behind, above, 
or on one or more performers. The projected video image may be actual or processed by 
video signal processing (for example, changing color maps, contrast, solarization quantiza- 
tion thresholds, etc.) which in turn may be controlled by control signals generated in real- 
time by the performers. In this manner, one or more performers may interactively perform 
with music, sound, and visual effect with a non-equilibrium chemical reaction environ- 
ment. 

It is also possible to numerically or electronically simulate the chemical dynamics on a 
computer, generating similar types of control signals, visual output, and interactive control 
capabilities. This methodology is discussed in more detail. Because numerical and elec- 
tronic simulation can generalize the process beyond physical limitations, in principal a 
broader range of interactive dynamics would be made possible by this method. However, 
the excitement and charm of interacting with a live chemical process is difficult to entirely 
replace with a computer program. 

Photoacoustic sources 

Photoacoustic phenomena is a relatively new area of study. Although most of the gathered 
knowledge and work in progress is largely oriented to probably inapplicable areas relating 
to, for example, non-destructive testing, there are a few phenomena, such as light stimulat- 
ed acoustic emissions and the modulation of light through vibrating transparent or translu- 
cent materials that can be developed for musical purposes [Lusher; Murphy et. al.; Bicanic, 
Dane]. The invention provides for the incorporation of these, particularly in that light can 
be used as part of performance and visually recorded material. 

Figure 48 illustrates example optical measurements of photoacoustic phenomena in appli- 
cable materials 4900, 4910 which may be converted to electrical signals, and an example 
electro-acoustic measurement of photo-induced acoustic phenomena in applicable materi- 
als 4920. Example photoacoustic materials 4900, 4910, 4920 may be in gas, crystal, liquid 
crystal, plastic, elastic, fluidic or other forms. 

For material 4900 which emits light in response to acoustic vibration, a light sensor 4901 
may be used to recover the light emission event. For material 4910 which modulates light 
in response to acoustic vibration, a light sensor 4901 may be used to recover light provided 
by a light source 4902 which is directed through the material 4910. For material 4920 which 
emits acoustic vibration in response to light, an electro-acoustic sensor 4921 may be used 
to sense acoustic vibration emitted in response to one or more appropriately positioned light 
sources 4902a, 4902b. 

In the above, it is noted that ultra-sonic vibration, even up to a few hundred Khz, is still 
potentially useful as these signals may be pitch-shifted or heterodyned down to audio rang- 
es. 



WDW 001/01 US 



Electronic/Numerical Dynamical System and Relational System 
Simulation 

Electronic and/or numerical algorithm methods may be used to implement mathematical 
dynamical models including mechanical vibration, fluid mechanics, stellar evolution, bio- 
logical processes, etc. as well as abstract non-equilibrium, fractal, and chaos process mod- 
els. Such methods are already in place in the synthesis of musical sound vibration processes 
modeling conventional musical instruments, for example, in model-based sound synthesis 
as used in the Yahama VL1 . 

Because numerical and electronic simulation can generalize the process beyond physical 
limitations, in principal a broader range of interactive dynamics and real-time measure- 
ments of them would be possible as compared to that which could be obtained in the real- 
world under realistic conditions. Further, electronically or numerically modeled processes 
may be time-scaled so as to produce audio frequencies or more slowly evolving control sig- 
nals. The invention provides for the use of such electronic and/or numerical algorithm 
method so as to implement mathematical dynamical models of adapted real-world or ab- 
stract processes. Incoming control signals can be used to select and/or affect the structure 
and/or parameters of the modeled dynamics and/or relations, and the modeled dynamics 
and/or relations may be used to create any one or more slowly varying outgoing control sig- 
nals, visual image signals, or direct audio signals. 

Examples of abstract processes may include interactive navigation through a fractal struc- 
ture, the fractional integration of an audio frequency square wave as it evolves into triangle 
and parabolic waveforms, etc. Examples of real-world models rich in semiotic value for 
performance may include adaptations of interactive control of galactic interaction dynam- 
ics, language models, etc. as well as the use of literary plots, classical mythologies, etc. 
which have been used by composers for centuries (i.e., Monteverdi's Orfeo, Strauss' Elec- 
tra, Stravinsky's Odepus Rex, etc.) 

Environmental 

Earlier instrument elements and instrument entities associated with environmental aspects 
of stages, rooms, and the outdoors were described. Examples of this include the tracking of 
the position and/or motion of performers, the tracking of artificial fog cloud migration, 
room internal and outdoor meteorology, and audience motion activity. As indicated in 
those discussions, these may be used, to the extent artistically applicable, to generate con- 
trol signals for the control of note events, signal processing, lighting, and special effects. 

Generalized Instrument Interfaces 

Referring to Figure 1 , it is recalled that the invention provides for both instrument entities 
100 and signal routing, processing, and synthesis entities 120 to be fitted with compatible 
electrical interfaces, termed generalized instrument interfaces or (or more concisely, gen- 
eralized interfaces) 110, which can exchange any of the following: 

• incoming electrical power (111) 
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• outgoing control signals from switches, controls, keyboards, sensors, etc., 
typically in the form of MIDI messages but which may also include contact 
closure or other formats (112) 

• control signals to lights, pyrotechnics, or other special effect elements within 
and/or attached to the instruments, said signals being either in the form of MIDI 
messages, contact closure, or other formats (113) 

• outgoing audio signals from individual audio-frequency elements or groups of 
audio-frequency elements within the instruments (114) 

• incoming excitation signals directed to individual audio-frequency elements or 
groups of audio-frequency elements within the instruments (115) 

• outgoing video signals (such as NTSC, PAL, SECAM) or image signals sent from 
the instrument (116) 

• incoming video signals (such as NTSC, PAL, SECAM) or image signals sent to 
the instrument for purposes such as display or as part of a visually controlled 
instrument (11 7). 

The interfaces may be realized by any one or more of: connectors, cables, fibers, radio 
links, wireless optical links, etc., individually, in combinations, or in or sequences of these. 

In most envisioned realizations this interface would be involve one or more connectors fit- 
ted with driving and/or receiving electronics, and the connectors on instrument entities 100 
and signal routing, processing, and synthesis entities 120 would be connected by a plurality 
of wires in either balanced or unbalanced transmission mode. Alternatively one or more 
coax cables, fiber optic cables, radio links, wireless optical links, etc. may be used to re- 
place part or all of the plurality of wires. Any of these approaches may use any of a variety 
of multiplexing techniques [frequency-modulated and/or phase-modulated and/or ampli- 
tude-modulated carrier, wavelength-division, time-division, carrier-less constellation syn- 
thesis (such as CAP), statistical, etc.) individually or in combination to reduce the number 
of partitioned physical signal channels (wires, fibers, radio channels, wavelengths, etc.]. 

When these generalized interfaces are realized via one or more physical cables (electrical, 
optical, etc.), some realizations may use a single connector for fully spanning generalized 
applications while other realizations may consist of an ensemble of connectors in a func- 
tional split so as to handle particular organization, expansion, and/or evolutionary needs. 

Figure 49 shows how generalized interfaces can be built in whole or via separable parts 
which may be used selectively as needed or appropriate. In a single connector all-purpose 
approach, a single cable 4901 carries all the signals allowed for in the generalized interface 
up to a predefined number of maximum instances. Referring to Figure 1, all instrument en- 
tities 100 and signal routing, processing, and synthesis entities 120 would provide mating 
connectors to that of 4902. In one implementation, the cable 4901 and connector 4902 pro- 
vide a relatively large number of separate interconnecting wires, for example twenty to thir- 
ty. Alternatively, in a future preferred arrangement employing then low-cost standard 
signal multiplexing and/or directional multiplexing techniques, this cable 4901 could be a 
simple one or two fiber optic or coax cables potentially supplemented with two to three 
power conductors; alternatively, if any coax cables are used to carry signals they can also 
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be used to simultaneously carry power on the same conductors. Further, if each instrument 
is able to provide its own power by means of wall plugs and/or batteries, the aforemen- 
tioned implementations need not include any power carrying capabilities and associated 
conductors. In this arrangement it will be further possible to use radio and/or optical wire- 
less channels to carry the signals among instrument entities 100 and signal routing, process- 
ing, and synthesis entities 120. In this arrangement then, a common multiplexed incoming 
and outgoing signal, potentially itself directionally multiplexed on the same channel, can 
be carried interchangeably by optical fiber, coax cable, wireless radio, or wireless optical 
transmission mediums. 

Alternatively, it is possible to functionally partition the generalized interface into standard- 
ized component interfaces which may be served by separate connectors. A multi-connector 
"Hydra" cable can be used to provide selected groups of two or more of these standardized 
connectors, including a "fully-populated" Hydra cable with all the defined connectors. If 
only one connector of the several defined ones is needed, then a single connector cable may 
be used if desired; for this reason, it may be desirable to assign connectors to the functional 
partitions which are standardly available on mass-produced single connector cables. Figure 
49 shows an example of this arrangement. A fully-populated Hydra cable 4911 may include 
the following partition and assigned connectors: 

• Two "single-channel" group audio outputs (unbalanced, dedicated ground) 4912; 
for example, a TRS male plug. 

Ten or twelve multi-channel audio outs (unbalanced, dedicated ground) and audio 
power 4913; for example a 13-pin DIN male or HDB 15-pin VGA male (with 14 
pins actually populated and connector shells interconnected). 

One video out channel and one video in channel (unbalanced, dedicated ground) 
and incoming video power 4914; 6-pin DIN male. 

• One balanced MIDI out channel and incoming MIDI power (on non-MIDI pins) 
4915; 5-pin DIN male. 

• balanced MIDI in channel and incoming controlled element power (on non-MIDI 
pins) 4916; 5-pin DIN male. 

• Six excitation drive channels (unbalanced, dedicated ground) 491 7; 8-pin DIN 
connector (only six pins need if the signal routing, processing, and synthesis 
entity 120 creates the relatively high-power drive signals, while eight pins allow 
provision of power to the instrument entity 100 for internal high-power signal 
generation). 

Further, any connectors not served by a given Hydra cable and/or expansions to support ad- 
ditional channel-carrying needs may be supported with additional cables: 

It is understood that the aforementioned as explained and illustrated in Figure 49 are merely 
examples; other arrangements are possible. 
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7 Signal Routing, Processing, and Synthesis 

The general principals for the architecture of the signal routing, processing, and synthesis 
entity 120 as provided for by the invention include all or a significant number of the fol- 
5 lowing: 

flexible multi-channel handling of audio, control, and video signals 

• a hierarchically modular control and control signal routing structure 

• course to very fine-grain control signal routing (for example, in the context of 
^ MIDI, routing at the MIDI port level, routing the MIDI channel level, and routing 

at the individual note number and continuous controller number levels) 

• the incorporation of mixing in audio routing and message merging and polyadic 
operations in control signal routing 

1 5 • control signal extraction from audio and/or video 

• audio signal, control signal, and potentially video signal processing 
q • audio signal, control signal, and potentially video signal synthesis 
|g • real-time control signal/event replay 

-u all under extensive real-time control. 



r 7.1 Audio signal routing 

8 



m 



Audio signal routing is provided for in the invention by both switching and mixing func- 
tions. Switching functions may be realized as stored program cross-bar switches. Mixing 
functions may be provided in the form of possible multiple-input multiple-output mixing 
matrices and an additional final mixing stage may include some dedicated signal process- 
ing functions. Mixing functions provided for in the invention are advantageously controlled 
in real-time by control signals. 

Functional examples of the functionality provided for in said multiple-input, multiple out- 
put mixing matrices is that of the Sound Sculpture model Switchblade MIDI-controlled 
mixer (but empowered with a significantly larger number than two MIDI continuous con- 
troller inputs) or the Peavey PM-8128 (but provide with additional inputs and outputs). 
35 Functional examples of the functionality provided for in said final mixing is that of the Ya- 

hama DMP MIDI-controlled mixer models, particularly the DMP 9-16 (but with additional 
presets). In the invention, the mixing and switching functions are preferably an integrated 
component within a larger-scale hardware and software construct rather than an off-the- 
shelf module. 
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7.1.1 General audio switching and mixing 



Referring to Figures 1 -2, input signals directed to audio routing and mixing may include 
the audio outputs from the instrument entities 100, outputs from audio signal processing el- 
ements 125, and audio signal synthesis elements 129a. Still referring to Figures 1-2, output 
45 signals directed from audio routing and mixing may include audio inputs to exciter ele- 
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merits within instrument entities 100, audio signal processing 125, control signal extraction 
128a, and the overall audio outputs to amplification and/or recording facilities. 

7.1.2 Multi-charm I Audio Signal Handling 

The invention provides for extensive support for and exploitation of multi-channel audio 
signals from instruments with multiple vibrating elements. 

Multi-channel transducers have been used in multiple-vibrating-element musical instru- 
ments; these uses appear to be confined to guitar synthesizer interfaces (as with the Boss 
GP- 1 0), individual adjustment of each vibrating element mix level (as with the Gibson Chet 
Akins guitar), and creation of panned stereo mixes (Biax pickup, Passaic synthesizer inter- 
face, Turner string pan-pot guitar, Van Halen-endorsed guitar with right/left switches for 
each string). These similar approaches may be generalized by a common diagram. Figure 
50 shows multiple vibrational elements with multi-channel transducers applied directly to 
stereo or multi-channel mix-down. Each of a plurality of vibrating elements 5001,1- 5001. n 
are is coupled by appropriate but separable means to each of a plurality of transducers 
5002,1 - 5002,n, each producing electrical signals which are applied to either single chan- 
nel or stereo mixing circuitry. In the case of the Gibson Chet Atkins electric-acoustic guitar, 
the mix is monaural and as such is equivalent to delivery only through output A 5004a, with 
no implementation of output B 5004b. In the case of the Biax pickup, the "pan" position in 
the stereo field is hard-set to either output A 5004a or output B 5004b. In the case of the 
'7.i Turner guitars, the "pan" position in the stereo field is set between output A 5004a or output 

5 B 5004b via adjustable potentiometers. 

Specifically the invention provides for bringing the signals from multi-channel transducers 
^5 5002,1 - 5002,n to individual signal processing stages 5005,1 - 5005,n before mixing, al- 

^ lowing far more extensive capabilities to be created. Figure 51 shows multiple vibrational 

7J elements with multi-channel transducers and individual signal processing prior to mixing. 

7 Figure 5 1 highlights the functional signal processing distinction from existing commercial 

q products generalized by Figure 50. This relatively simple conceptual (though potentially 

JO hardware and/or software intensive) change makes a number of extraordinary things possi- 

i ble: 

Conventional pitch-shifting signal processing can be used on each string signal 
to create: 

35 • "generalized pedal steel guitars" (augmenting or replacing mechanical pedal 

tuning changers with pedal, lever, spring-wheel, optical, or other electronic 
controls determining pitch shift amount) 

instantly retunable guitars (augmenting or replacing mechanical tuning 
changers such as the Hip-shot "Trilogy") 

• a true electronic simulation of so-called "multi-course" instruments (such as 
a 12-string guitar, mandolin, lute, etc.) where individual elements making 
up the "multi-course" are simulated using pitch shifting to create either 
octaves or slightly mis-tuned unisons 

45 • multi-key, multi-modal Indian sitars; here drone and sympathetic strings 

can be electronically retuned while playing, allowing a more flexible and 
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robust mix between Eastern (fixed tonality) and Western (modulating 
tonality) musical forms. 

• multi-key, multi-modal African mbiras, African koras, Japanese kotos etc.; 
here fixed pitch vibrating elements (tynes, strings) can be electronically 
retuned while playing, allowing a more flexible and robust mix between 
Eastern (fixed tonality) and Western (modulating tonality) musical forms. 

• spatial-spectral animated instruments where individual vibrating element sounds 
may be location modulated within a stereophonic or other spatial sound field 
(using low-frequency sweep chorusing, continuous auto-panning, etc.). 

• separate distortion circuits for each vibrating element, for example, to create: 

• powerful guitar chord sounds previously obtained only by multiple 
instruments or multi-track recording) 

• simulated sitar-bridge effects (using the methods to be described in 
conjunction with Figure 39) 

• finely frequency-equalized instruments where different frequency equalizations 
are applied to each vibrating element. 

• mixed timbre instruments where different signal processing methods are applied 
to each vibrating element. 

(More recently, a functionally limited - although very technologically progressive - version 
of the signal processing approach illustrated in Figure 52 has since appeared in the Roland 
VG-1 COSM guitar emulation product.) 

Figure 52 shows addition of a control signal extraction element to the arrangement of Fig- 
ure 51. As shown in Figure 52, the invention provides for signal outputs from the multi- 
channel transducer(s) to be fed in parallel to a control signal extractor 5010 so as to simul- 
taneously issue control signals SOU to any one or more of music synthesizers, signal pro- 
cessors, lighting, special effects, etc. The invention therein provides for an operating mode 
of control signal extraction where each individual vibrating element to have associated with 
it one or more individual control signals. The resulting system, then, can assign a given vi- 
brating element to individual signal processing, individual synthesizer voice, or both in 
combination. This permits a basic conventual instrument structure (such as a guitar, violin, 
steel guitar, koto, or mbira) and essentially conventional playing techniques to control an 
unprecedented rich range of sounds. (Even more recently, Roland has since announced a 
cable fan-out product allowing their VG-1 multi-channel signal processor to operate simul- 
taneously with their GP-10 and related guitar to MIDI interface products.) 

Further enhancements are also possible. For example: 

• In practice it may be desirable to have a different number of signal processors than 
vibrating elements. For example: 

• in generalized steel guitars, only a few strings at a time may actually be 
candidates for pitch shifting 
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• in spatial-spectral (panning, chorusing, etc.) animation, the actual number of 
animation channels need not match the number of vibrating elements 

in a highly functional system, several signal processors may be used in 
parallel for one or more vibrating elements. 

Figure 53 shows partial mix-downs of vibrating element signals fed to a number of signal 
processors 5005.1 - 5005m and straight-through paths 5007.1 - 5007. m en route to subse- 
quent mix-down. The (pre-signal processing) input mixer 5006 is used to route and/or mix 
various multi-channel transducer signals to a structured and controllable multi-channel 
mix. Several variations of this arrangement are suggested by Figure 53: 

• there can be zero, one or more straight paths 5007.1 - 5007. m. 

• there are a total of at least three signal processors 5005.1 - 5005.n and/or straight 
paths 5007.1 - 5007.m involved. 

• there is a minimum of one signal processor (or else the two mixers 5003 and 5006 
functionally collapse into one, functionally resulting in the arrangement of Figure 
50). 

although the interconnection details for connecting the synthesizer interface are 
shown, the synthesizer interface need not be included. 

Figure 54 shows another arrangement wherein a switching matrix 5008 is used in place of 
the input mixer 5008 to select which individual vibrating element 5001.1 - 500 l.n signal is 
assigned to which signal processor 5005.1 - 5005.k. This arrangement is particularly rele- 
vant to the generalized retuning of certain vibrational elements, as in adapted pedal steel 
guitar and adapted sitar described earlier. 

The invention also provides for several signal processors to be pooled and used in various 
parallel, series, or other topological interconnections serving one or more vibrating ele- 
ments. Figure 55 shows a more flexible example method for providing signal processors 
with vibrating elements' signals and other signal processor outputs via switch matrix, and 
additional partial mix-downs by replacing said switch matrix 5008 with an input mixer 
5006, 

The invention provides for any of the above systems to be integrated together into a com- 
mon system sharing a common configuration preset storage and recall facility. Figure 56 
shows configuration control of signal processors, mixers, any switch matrix, and synthesiz- 
er interfaces via logic circuitry and/or microprocessing. As illustrated in Figure 56, this is 
simply a matter of putting all or some combination of the mixers (5003 and/or 5006), switch 
matrices 5008, signal processors 5005.1 - 5005.k, and/or synthesizer interfaces, as relevant, 
under the control of logic circuitry and/or microprocessors 5009 which can provide such 
preset storage and recall functions. 

By combining the multi-channel signal handling with excitation, not only can individual 
vibrating elements be assigned to various signal processing and synthesizer controlling 
roles, but also individual vibrating elements can now be assigned feedback modes where 
selected vibrating elements can sustain vibration as if they were bowed, in an electric-guitar 
feedback arrangement, etc. Further, through use of additional switching, signal processing 
can be added to the feedback loop as discussed earlier, but on an individual vibrational el- 
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ement basis. Finally, since feedback arrangements tend to emphasize higher harmonics of 
vibration, and the dynamics of the relative levels of the harmonic mix can be varied dra- 
matically by touching elements or varying feedback characteristics (via signal processing 
in the excitation feedback loop), the invention provides for control signal extraction to be 
5 expanded to respond to details of the overtone content as discussed later. 

Figure 57 shows a very general combined environment for multi-channel signal processing, 
mixing, excitation, and program control of overall configuration. In Figure 57 the general 
combined environment incorporates a plurality of separate feedback loops for each vibrat- 
ing element, each loop featuring a loop signal processor 5021.1 - 502 Ln (which here could 
be as simple as a level control) which may be controlled by control signals as provided for 
in the invention. Many possible variations of this approach which may omit or simplify any 
of the elements shown in Figure 57 can be realized; for example the signal processors 
5005.1 - 5005.1k and 5021.1 - 502 Ln may be pooled together in their association with mix- 
ers 5006 and/or switches 5008 with said mixers 5006 and/or switches 5008 sending signals 
to the excitation drive amplifiers 5022.1 - 5022.n. 

7.2 Audio Signal Processing 

; % Many of the audio signal processing elements cited as 125 (Figure 1), 2211 (Figures 25- 

gj 26), 5005.1 - 5005.n/k (Figures 51-57), and elsewhere in this document can be adequately 

\2 realized by any number of the standard multi-function MIDI-controlled signal processing 

ru modules such as the Roland model RSP-550, Boss model SE-70, Ensoniq model DP/4, 

; p ART model SGE Mach-II, Digitech model GSP21, etc. In the invention, these signal pro- 

O cessing functions are preferably an integrated component within a larger-scale hardware 

f§ and software construct rather than an off-the-shelf module. The invention provides not only 

for the incorporation of these into the signal routing, processing, and synthesis entity 120 
(Figures 1-2) in the ways described below but also for additional audio signal processing 
methods which are not commercially available and which are described below. 

Spatially Distributed Timbre Construction 

Because of the extensive biaural capabilities of human hearing, stereo and other multi- 
channel sound fields can be used to create a number of musically useful timbral construc- 
tion ranging from the subtle to the powerful and the beautiful to the bombastic. 

Examples of this, commonly found, are stereo-output chorus, stereo-output flangers, ste- 
reo-output reverb, stereo-output echos, etc.; but the spatial construction of timbres may be 
carried far beyond these simple and now commonplace effects. The following discussion 
explains some example techniques; the role and value of these techniques are developed 
further in subsequent material following that below. 

Cross-channel modulated delay 

The invention provides for methods to enhance, and to more significantly increase the 
depth of, a stereo signal set source whose components have similar but slightly different 
timbres, particularly if the timbres are time- varying. Examples of such stereo signal set 
sources include the stereo outputs of traditional choruses, flangers, reverbs, etc., a pair of 
signal distortion elements with different characteristics, two harmonized synthesizer voices 
or pitch-shifter outputs, the separate outputs of a single two-oscillator synthesizer voice, 
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etc. Figure 58 shows a stereo-input, stereo output configuration of two monaural flange 
and/or chorus elements wherein the unaltered signal of each input channel is combined with 
a delay-modulated signal from the opposite channel. Each input 5801, 5802 is presented to 
a dedicated delay-modulation element 5809, 5810 and a dedicated output channel summer 
5 5831, 5832. Internally each delay-modulation element 5809, 5810 consists of a variable de- 

lay implemented in this example by changing the clock speed of a clocked delay line 5803, 
5804 by means of a variable clock oscillator 5805, 5806. The speed of the clocking oscil- 
lator is controlled by a variable low-speed modulating "sweep" oscillator 5807, 5808. Since 
the sweep oscillator's output is periodic, it is possible to implement the clocked delay line 

10 5803, 5804 not only by changing the sample rate but by loading fixed-rate samples into a 

ring buffer and reading out of the ring buffer at a rate set by the variable clock 5805, 5806. 
Other implementations of delay-modulation elements 5809, 5810 are also possible as 
known to those familiar with the art. The outputs 5821, 5822, respectively, of each delay- 
modulation element 5809, 5810 respectively, are directed to the summing elements 5832, 

1 5 5833, respectively associated with the opposite unaltered input channel 5802, 5801, respec- 

tively, producing stereo outputs 5832, 5831. The delayed signals 5821, 5822, may be 
summed at the summing elements 5832, 5833 in additive, subtractive or other phase-shaft- 

=«i ed or phase-dispersed relationships. In a preferred implementation the two sweep oscilla- 

;B tors 5807, 5808 operate at slightly different sweep frequencies and are free-running 

Sp (unsynchronized) if possible. 

\Z (It is noted that a similar, restricted version of this has since been incorporated as one of the 

~'f effector modes, namely "cross-over chorus", of the Korg model X5DR synthesizer module. 

j=j In the Korg implementation, however, the two sweep oscillators 5807, 5808 have been re- 

ig^ placed by a single sweep oscillator with two phase-locked quadrature, i.e., 90-degree phase 

; difference, outputs. 

jf: It is noted the above arrangement may naturally be extended beyond stereo to accommo- 

Y2 date additional input and/or output channels. The most general implementation would have 

~p N inputs, M outputs, N-l variable speed swept delays, and M summers with N inputs 

i JO summed with adjustable gains and/or phase relationships; simplifications of course are pos- 

U sible. One example application would include M-speaker (i.e., M=4 for quadraphonic) am- 

plification. Another example application with M=2 for stereo and N>2 similar signal 
sources would build an enhanced version of the sonic effect. 

* In the above it is noted that when pluralities of elements (for example, spatializer and dis- 

tortion elements) are cited, the elements in the plurality need not be identical in their type 
and/or parameterized settings. Further, various parameters of each of the elements (modu- 
lation speed, modulation depths, relative amplitudes in audio mixes, etc.) may be advanta- 
geously controlled in real-time by control signals for expression (from instrument entities, 
foot controllers, etc.), further correlation with the signal source (for example, using enve- 

^ lope extraction control signals) or further levels of animated enhancement (employing ad- 

ditional sweep oscillators, envelope generators, etc.). 

7.2.1.2 Multi-Level Stereo Chorused Distortion of Monaural Sources 

The invention provides for creating a similar-signal stereo signal set from two distortion 
45 sources and presenting it to cross-channel modulated delay to synergistically transform a 

relatively spectrally dull signal, particularly a time varying one, into a very rich powerful 
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sound. Figure 59 illustrates a combination of a spatialized effect, two distortion elements, 
and a stereo (N=M=2) cross-channel modulated delay. In this example arrangement, the in- 
put signal 5900, which may be from a group pickup as shown, individual vibrating element 
pickup, audio signal synthesis element, microphone, etc., is applied to a stereo-output spa- 
5 tializing effect 5901 such as a stereo output chorus, flanger, etc. The resulting stereo signals 

5902, 5903 are applied to distortion elements 5904, 5905. Fluctuations in the waveshape of 
the applied stereo signals 5902, 5903 cause significant dynamic timbral shifts from the dis- 
tortion elements 5904, 5905; such fluctuations in the applied stereo signal waveshape may 
originate from the original signal 5900, from any one or more of amplitude animation, 
10 phase animation, delay accumulation, dynamic waveshaping processes, etc. in the spatial- 

izer 59 01, or both in combination. The distortion elements 5904, 5905 produce a rich sim- 
ilar-signal stereo signal set 5906, 5907 which is then presented to a N=M=2 cross-channel 
modulated delay, resulting in stereo output signals 5909a, 5909b. 

For a single input channel 5900, the invention provides for the expansion of such an ar- 
rangement to include additional processes to build an enhanced version of the sonic effect. 
For example, an N-output version of the spatializer 5901 (which may, for example, be im- 
plemented internally by two or more simpler spatializers in parallel, hierarchical, or other 
P interconnection topologies) can be used in conjunction with N distortion elements in an N- 

input (N>2) M -output cross-channel modulated delay replacing 5908. 

M For multi-channel signal sources, the invention provides for each signal to be handled by a 

TO dedicated spatializer and several possible subsequent processing arrangements. As one ex- 

=F ample, assuming K input channels, selected outputs of each of the K spatializers may be 

y mixed and presented to N (N being two or more) distortion elements which in turn are pre- 

y sented to an N input, M output cross-channel modulated delay replacing 5908. In another 

example, no pre-distortion mixing is used but rather each spatializer is provided with its 
JTJ own collection of two or more distortion elements; the collection of all outputs of these, 

|T which are of number J not equal to N, may be matrix-mixed to form N mixed outputs which 

k Q are applied to an N input, M output cross-channel modulated delay replacing 5908. In an- 

(g> other example, no pre-distortion mixing is used but rather each spatializer is provided with 

M its own collection of two or more distortion elements; the collection of all outputs of these, 

which are of number N, may be directly applied to an N input, M output cross-channel mod- 
ulated delay replacing 5908. Other arrangements similar in form and spirit are clearly pos- 
sible. 

3^ In the above it is noted that when pluralities of elements (for example, spatializer and dis- 

tortion elements) are cited, the elements in the plurality need not be identical in their type 
and/or parameterized settings. Further, various parameters of each of the elements (modu- 
lation speed, modulation depths, relative amplitudes in audio mixes, distortion parameters, 
etc.) may be advantageously controlled in real-time by control signals for expression (from 

^ instrument entities, foot controllers, etc.), further correlation with the signal source (for ex- 

ample, using envelope extraction control signals) or further levels of animated enhance- 
ment (employing additional sweep oscillators, envelope generators, etc.). 

7.2.1.3 Location Modulation 

45 Location modulation has been commercially available in the form of "auto-panning" where 

an audio source is periodically panned back and forth between two stereo outputs. The in- 
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vention provides for limiting periodic auto-panning of monaural sources sounding in isola- 
tion to be typically most effective when the degree of panning is limited and the modulation 
rate is low (as extreme settings of modulation depth and speed are typically not as widely 
musically useful). Under these conditions in a stereo sound field a signal source takes on 
an animated character but yet is not so blatantly spectrally modified as it is in chorus and 
flanging effects. The invention also provides for wider ranges of depth and speed to be used 
in the context of multi-channel auto-panning, discussed next, and layered signal processing 
discussed below and already touched upon in the discussion associated with Figure 39. 

The invention provides for multi-channel versions of auto-panning. In layered signal pro- 
cessing, such as that discussed in the context of Figure 39, auto-panning contributions work 
best within the invention if modulation sweep oscillators operate at different (typically only 
slightly different) modulation speeds. In the context of multi-channel signals provided by 
individual vibrating elements of an instrument entity 100, this unsynchronized configura- 
tion of modulation sweep oscillators can create inhomogeneous "bunching" effects when 
many modulation sweep oscillators take on identical or nearly- identical values in extreme 
ranges of the modulation sweep. 

^ The invention provides for a much more homogeneous method for multi-channel periodic- 

^ sweep auto-panning, namely that of arranging the signal pan images in a phase-staggered 

constellation swept by a single modulating sweep oscillator. A simple example is that of 
U stereo cross-panning where two input signals pan between stereo speakers in synchronized 

lU complementary directions. Another example is that of staggering the phases of a multiple 

=F phase output modulating sweep oscillator in some preassigned arrangement, such as offset 

O from each other by a common phase-offset value. This may be used to pan the sounds from 

y> each individual vibrating element so that the individual panned sound images follow one 

;^ another between two speakers. Similar methods can be used if there are more outputs (for 

Y~i example, quadraphonic, hexaphonic, octaphonic etc. speaker installations aligned in a 

plane or in 3 dimensions); here N input, M output mixers can be controlled by one or more 
lp single or multiple-phase output modulating sweep oscillators. 

® Control- signal invoked transient "one-shot" panning effects may also be obtained from 

- M commercial mixer products that feature a fade-time transient between pre-programmed am- 

plitude settings (such as the Yahama DMP series and Sound Sculpture Switchblade series). 
The invention provides for such transient effects to be used as a compositional element in 
music or a metaphorical or semiotic element in audio and/or audio- visual aspects of perfor- 

^ mance. In particular limited-duration panning trajectories of arbitrary nature, each affiliat- 

ed with one of several individual sound sources, may be made to simultaneously and.or 
sequentially follow a predefined relative dynamical pattern. This can be used as a contra- 
puntal element in melody or abstract musical forms. It can also be used to create plot events 
in a composition or performance, such as in a musical composition, dance composition, or 

^ play concerning or involving the spatial interaction of bird sounds. 

7.2.1.4 Other Spatially Distributed Timbre Methods 

Several other aspects of the invention to be presented below in other contexts also may be 
used to create spatially-distributed timbral realizations; their use as general audio signal 
45 processing elements 129a in this fashion is provided for as part of the invention. 
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One aspect of the invention which may be used for spatially-distributed timbral realizations 
is: the two-input or multiple- input versions of the octave cross-product chain described lat- 
er on in the context of audio signal synthesis waveshaping. As described there, this tech- 
nique results in a number of parallel signal outputs with widely differing spectral contents 
5 and spectral animation features, and the animation features slow to a halt when all funda- 

mental and overtone frequencies of the two input signals are brought into fixed integer and 
small integer-ratio multiplicative relationships. The aforementioned characteristics of the 
multiple outputs lend themselves to spatially-distributed timbral realizations since mixing 
of the outputs can partition the frequency content and animation features differently be- 

1 0 tween final mix-down outputs. The invention provides for this method to be used as a signal 

processing technique. In one example usage, a pitch-shifter, swept variable delay, etc. is 
used to construct a derivative frequency and/or phase shifted signal (the characteristics of 
which may be controlled by control signals for expression) from an original signal. The 
original and derivative signals are then fed into the octave cross-product chain to produce 

1 5 often spectacular spatially-distributed timbral realizations. 

Another aspect of the invention which may be used for spatially-distributed timbral real- 
izations is multi-channel waveshaping where a signal source is provided to a plurality of 

O waveshapers each of which may be controlled by control signals. Each waveshaper output 

jO may then have differing frequency content and animation features which thus lend'them- 

]4 selves to spatially-distributed timbral realizations in a multi-channel (stereo, quadraphonic, 

|~ etc.) partition or mix-down. Of particular interest is the use of hysteretic waveshaping, de- 

*!! scribed later, which creates a wide range of spectral differences as the input waveform 

% and/or hysteresis parameters change over time. 

B| Another aspect of the invention which may be used for spatially-distributed timbral real- 

; ^ izations is the use of later described layered audio signal processing methods. The invention 

J ~; does this by providing for each audio signal processing layer to be allocated a different pro- 

[ J portion to each final mix-down output channel. These allocated mix proportions may be 

[Z varied over time by control signals. 

~ 7.2.2 Multi-channel Audio Signal Handling 

The invention provides for flexible homogeneous and inhomogeneous signal processing of 
multi-channel audio sources. Such multi-channel audio sources may for example include, 
referring to Figure 1-2, a multiple vibrating element instrument entity 100, multiple in- 
35 stances of audio signal synthesis elements 129a with single or multiple output-channels au- 

dio signal synthesis elements. 

Several signal processing methods involving multi-channel signal sources have already 
been discussed thus far, particularly those in the previous few sub-sections. The invention 
further explicitly provides for dedicated, shared, or combined arrangements for audio sig- 
nal processing elements within the signal routing, processing, and synthesis entities 100 as 
shown in Figures 53-57. In particular, each signal from a multi-channel source may be han- 
dled differently with at least some of these signals processed by one or more audio signal 
processing elements 129a of identical, similar, or differing function. Conceptually, the 
^ most flexible of these are embodiments where input mixing 5006 is used to share the inputs 

and outputs of a pooled collection of signal processors as in Figures 55-57. The audio signal 
processing elements 129a may be any one or more of conventional signal processing func- 
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tions provided by commercial products (chorus, flange, reverb, distortion, delay, filtering, 
equalization, etc., individually or in combination) as well as any one or more of the inven- 
tion's novel audio signal processing methods described thus far and below. 

5 7.2.3 Bass N t Derivation 

The invention provides for the derivation of bass notes from signal sources. This is partic- 
ularly relevant in the invention where signals from selected vibrating elements are used to 
create bass notes. The created bass notes may be heard in parallel with the original pitch of 
the signal (each pitch may be subject to different signal processing) or in replacement of it. 
10 In many cases this completely eliminates the need for bass accompaniment in a perfor- 

mance situation at the potential expense of melodic freedom of the bass line. 

The invention provides for at least three methods of bass note derivation which may be used 
individually or in combination. 

1 5 One of these methods is the use of control signal extraction to derive note events to run a 

bass note audio synthesis element (for example, a conventional audio synthesizer module 
transposed down one or more octaves or other large interval). If the bass interval is not al- 
ways to be fixed, pre-programmed note transpositions reflecting desired harmony and/or 
player-controlled changes in pitch- shift interval may be used individually or in combina- 
i|p tion. This audio synthesizer method allows a wide range of sounds to be used but can be 

!=£ limited in how the bass note expression can be controlled from the original signal source, 

jy One solution to this provided for by the invention is the use of overtone parameter tracking 

=p in the control signal extraction; these additional parameters may be used to shape the syn- 

=3 thesized sound though varying parameters in the synthesis processes and/or by varying sub- 

55 sequence signal processing parameters. 

q Another of the methods is through the use of conventional pitch shifters. If the bass interval 

Id is not always to be fixed, so called "intelligent-harmony" pitch shifters (such as the Dig- 

N itech model IP-33B) and or player-controlled changes in pitch-shift interval may be used 

individually or in combination. The use of pitch-shifting allows for nuances of the original 
§P signal source to be carried through but may suffer from delayed response, glitch, phasing, 

l= * "Darth Varder," or other undesirable or limiting artifacts. 

Yet another method, should the bass interval always be related to the source pitch by oc- 
taves, the invention provides for an adaptation of the novel octave divide method used in 

35 the Boss OC-2 "Octaver" pedal. Although this technology does have glitching and mono- 

phonic limitations as described below, it works very well in responding to amplitude enve- 
lope attributes of the signal source. As is evident from the publicly available published 
service note schematic and usage of the device, each octave signal is created by frequency 
dividing the original signal (for example by means of a toggle flip-flop), scaling its ampli- 

40 tude by the instantaneous amplitude of the source signal (for example, through use of an 

envelope follower and a gain-control method), and combining this with a bit of the original 
signal to create a richer resulting overtone result. The unit suffers from the fact that har- 
monically rich signals often confuse the frequency dividers resulting in a very glitchy bass 
signal. Further, the method is monophonic; the playing of two notes at once processes only 

45 one bass signal, and usually an unusably unstable one. The invention provides for the 

glitch-free adaptation of the OC-2 technology to multiple vibrating element instruments by 
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dedicating a specific low-pass filter and an allocated (or allocatable) OC-2 divider or divid- 
er chain to each selected vibrating element. In particular, the incoming individual vibrating 
element signal is low-pass filtered to greatly attenuate frequencies above the maximal fun- 
damental frequency to be recognized by the arrangement (this maximal value may, in some 
5 circumstance be high enough to support unfretted string "chime" harmonics and the like). 

The combination of applying each instance OC-2 technology to a single vibrating element 
together with a highly emphasized fundamental frequency eliminates the glitching and 
monophonic limitations. The invention provides for a plurality of the described OC-2/filter 
arrangements, numbering for example three for a guitar, to be allocated to specific vibrating 
10 elements (fixed by design, selectable via stored program control, etc.). Further, the inven- 

tion provides for the use of this technology should bass notes need to be non-octave in re- 
lation to the original signal: the nearest octave note can be generated by the OC-2/filter 
approach and an allocated pitch shifter may be used to make relatively smaller pitch chang- 
es, recognizing that smaller shift intervals tend to have less artifacts. 

15 

7.2.4 Layered Audio Signal Processing 

The invention provides for the layering of multiple audio signal processing paths driven 
p from one or more shared sources and partitioned or mixed down to two or more output 

=0 channels. Because this may be viewed as a superposition of several signal processing paths, 

this will be referred to as "layered audio signal processing." One example of this has al- 
ii- ready been presented in the discussion relating to Figure 39; here each layer is responsible 
^ for emulating a separate sympathetic string effect. As each layer is in the Figure 39 example 
;t essentially identical, the layers may be called "homogeneous." (Some examples of homog- 
\i enous signal processing have since been devised, for example the "PentaChorus" preset of 
25 the Roland RSP-550 signal processing module.) In contrast to homogeneous layered signal 
processing, Figure 60 illustrates separable examples of inhomogeneous layered signal pro- 
lyl cessing. The examples of Figure 60 may be used as shown, with selected omissions, or as 
]=& an archtype for similar constructions as provided for in the invention. In one example a 
=p group pickup signal 6001 is applied to a distortion element 6011 and a compression element 
3D 6013. Each of the output's signals 6104, 6005 as well as the original signal are provided to 
I s * separate spatializers 6012 - 6015 (for example, chorus, flange, reverb, etc.) which are then 
mixed down into a stereo signal 6021a, 6021b by an output mixer 6020. The invention pro- 
vides for the substitution of any of the elements 6011, 6013, 6012 - 6015 with other types 
of audio signal processing elements as well as inclusion of additional layers. As expansion 
35 of the example, individual vibrating element signals from vibrating elements sharing the 
aforementioned group pickup can also be used to create additional layers. For example, an 
individual bass string signal 6002 (for example, the 5th and or 6th string of a guitar) may, 
in parallel, be processed by a pitch- shifter, OC-2/filter, etc. 6016 to create a bass note pitch 
signal 6006 which, in turn, may be presented to a separate spatializer 601 7. Further, another 
40 individual string signal 6003 can be processed in a similar fashion but replacing the pitch- 
shifter with an emphasis effect for emphasizing a particular melody or note in a chord. 

Because of the larger number of sonic sources that can be statically distributed in the sound 
field, the invention provides for the use of location modulation with a wider range of per- 
^ missible modulation rates and modulation depths as extremal location modulation behavior 

is only part of the overall spatial sonic structure. 
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The invention also provides for the use of layered audio signal processing in the creation 
of spatially-distributed timbral realizations. One example of this would be providing a ded- 
icated stereo chorus to each of the six individual string signals of a guitar as well as a sev- 
enth stereo chorus to the group pickup signal, setting each chorus sweep rate slightly 
5 differently and summing the seven stereo outputs into a single stereo mix; this is in fact an 

example adaptation of the principals illustrated in Figure 60. Another example is that of the 
cross-product octave chain to be described later. 

The invention provides for the use of waveshaping techniques, particularly those which can 
be varied in real-time by control signals and/or hysteretic waveshaping techniques, as sig- 
nal processing elements. The invention also provides in general for the separate and/or co- 
ordinated control of parameters involved at each audio signal processing layer by means of 
general control signals. 

Envelope-controlled time and pitch modulation 

The invention provides for the modulation of the delay time of a variable delay line by a 
control signal corresponding to the amplitude envelope of the delayed signal or an associ- 
ated signal. This causes a tape-recorder speed instability effect correlated to the transient 
character of the reference signal amplitude envelope; more precisely the pitch changes with 
the time derivative of the amplitude envelope. The invention also provides for the substitu- 
tion of a variable pitch shifter controlled by the time derivative of the same control signal; 
this arrangement produces roughly the same effect. In either implementation the control 
signal may be first warped by an emphasis non-linearity, control signal delay, and/or other 
processing functions. The result can be used in soloing as a climactic effect or in modera- 
tion for a transient enhancement. The invention also provides for envelope control of pitch- 
shifting without time-differentiating the control signal 

Resonant distorting delays 

The invention provides for the sitar-like sympathetic/buzz emulation utilizing short high- 
resonant delays as described in association with Figure 39 to be used as a more general sig- 
nal processing element. This is particularly useful if parameters of the configuration, such 
as degrees of resonance, degrees of clipping, and modulation depths, can be varied in real- 
time by control signals. 

Hysteretic waveshaping and distortion 

Hysteresis occurs to some extent in overdriven tube amplifier output transformers due to 
the natural hysteretic properties of the materials used to make the transformer core. Hyster- 
esis effects in waveform distortion can create valuable amplitude- varying... The invention 
provides for generalized models of hysteresis to be used as a waveshaping technique, and 
as such a signal processing technique, with parameters of the hysteresis action variable in 
real-time via control signals. 

Traditional hysteresis curves for transformers, gears, pseudo-elastic deformation, etc. are 
well known (see for example [Visintin]. Figure 61 illustrates an example of a generalized 
hysteresis model construction as provided for by the invention. The input/output graph 
45 shows example symmetric curves that are linear 6102, superlinear 6103, and sub-linear 

6104 along with the time/amplitude oscillograph of an example applied waveform 6110. 
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Other types of symmetric or non-symmetric non-linearities may also be used. A time-de- 
rivative operation on the applied signal waveform 6110 followed by sign detection reveals 
whether the applied signal waveform is at any instant increasing or decreasing. As an ex- 
ample, the applied signal waveform 6110 would be applied to one non-linear warping func- 
5 tion such as 6103, 6104 when increasing and the other when decreasing, resulting in the 

waveform made of segments 6113, 6114 rather than the waveform 6112 that would have 
been created by the linear curve 6102. In order to allow the applied input signal to vary in 
amplitude and still maintain continuity of the waveform, the invention provides for the 
warping non-linearities to be themselves adaptively scaled or otherwise altered based on 
10 amplitude information from the current and previous direction reversals, moving average 

of waveform area or waveform power, etc. The invention provides for aspects of the hys- 
teresis process, such as curve shapes and degrees of dependency on waveform history, to 
be varied in real-time by control parameters. 

^ _ Hysteretic waveshaping can be of use in layered audio signal processing and spatially-dis- 

tributed timbral realizations which have been described above. 

7.3 Audio Signal Synthesis 

,q Referring to Figure 1 -2, the invention provides for the inclusion of audio signal processing 

elements 125 with the signal routing, processing, and synthesis entity 120. These may in- 
M elude conventional MIDI synthesizers, analog synthesis elements, or other technologies 

vJ particular to the invention such as cross-product octave chains, hysteretic waveshaping, 

■F vowel sound synthesis, etc. The invention provides for as many parameters as possible to 

y be variable in real-time by means of standardized control signal formats. The invention also 

25 provides for the synthesis of vibrating element feedback sounds controlled by control sig- 

~ nals and pitch sampling as may be adapted from the Boss DF-2 "Distortion/Feedback" 

JH product. 

^ 7.3.1 Spatially Distributed Timbre Construction 

SO It is possible to create spatially distributed timbre realizations as part of the audio synthesis 

l= process as well as by subsequent signal processing (cross-channel modulated delay, multi- 

layer chorused stereo distortion, phased multi-signal constellation location modulation, 
etc.) as described earlier. The invention provides for spatially distributed timbre realiza- 
tions within synthesis by a variety of methods. One method, found in many commercial 
35 synthesizer modules (such as the Korg M3-R, Korg X5DR, and Kawia K4-r, for example), 

is for the synthesizer voices themselves to involve multiple parallel oscillators and/or sam- 
ple-players delivered in the stereo or other multi output form. This sub-section discusses 
two other methods provided for by the invention. 

Cross-product octave chain 

The many times aforementioned cross-product octave chain involves two or more octave 
divider chains whose corresponding outputs are multiplied together, with all resulting out- 
puts summed together by a multiple output mix-down mixer. The cross-product technique 
results in a number of parallel signal outputs with widely differing spectral contents and 
spectral animation features, and the animation features slow to a halt when all fundamental 
and overtone frequencies of the two input signals are brought into fixed integer and small 
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integer-ratio multiplicative relationships. The aforementioned characteristics of the multi- 
ple outputs lend themselves to spatially-distributed timbral realizations since mixing of the 
outputs can partition the frequency content and animation features differently between the 
final mix-down outputs. The invention provides for the incorporation of cross-product oc- 
tave chains in audio single synthesis. 

Figure 62 shows an example implementation of a cross-product octave chain particularly 
suited to low cost implementation with logic chips or simple DSP program loops. Two in- 
put signals 6201a, 6201b are applied to optional comparators 6202 which convert the ap- 
plied signals into two-value waveforms. These are applied to a chain of octave dropping 
elements 6203 which here can be implemented in isolation as a chain of toggle flip-flops 
and in aggregate as a binary counter. The depth of the chain can include many levels with 
three to six a useful number of levels. Each of the resulting output signals at corresponding 
levels are multiplied together by multipliers 6204. Signal analysis of the truth table for var- 
ious logical operations show that for applied square waves an EXCLUSIVE-OR operation 
acts exactly as a unity-gain multiplication (while an AND function acts as a unity-gain mul- 
tiplication added to half-amplitude versions of the two applied square waves), so the mul- 
tipliers 6204 may be realized by EXCLUSIVE-OR operations. This amounts to less than 
$1US worth of chips in hardware and a small amount of code in software. All outputs are 
provided to an output mixer 6205 which produces at least stereo outputs 6206a, 6206b as 
well as potentially other mix-down outputs 6207.1 - 6207m and which in a preferred em- 
i'i bodiment may be adjusted in real-time by control signals. 

v i The invention provides for alternate implementations, fro example omitting the compara- 

■ II tors 6202, implementing the octave drop functions 6203 with pitch shifters or OC-2/fiiter 

2| technology, and/or implementing the multipliers 6204 with VCAs or 4-quadrant multiplier 

5 " operations. The invention also provides for expansions to include more than two octave 

q chains. 

rj In the context of audio signal synthesis, the applied signals 6201a, 6201b may be generated 

l Z by two oscillators within a single synthesizer voice; these oscillators may be relatively 

% tuned in unison, octaves, or near-consonant intervals for basic operation, and one of the os- 

\2 dilators may be continuously swept through a range of pitches to create huge audio dis- 

plays of pleasing spectral complexity, 

7.3.1.2 Multi-channel Waveshaping 

35 The invention also provides for spatially distributed timbre realizations through use of par- 

allel or complementary modulations of a plurality of waveshaping operations by control 
signals. The outputs of the plurality of waveshapers are then mixed into a stereo or multi- 
channel output mix. 

40 7.4 Control Signal Routing 

The invention provides for extensive control capabilities and as such requires sophisticated 
control routing, processing, and stored program organization. The capabilities for this pro- 
vided by the invention are described in the following sub-sections. To illustrate essential 
45 capabilities the discussion below is stated in terms of commonly appreciated MIDI messag- 
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es and conventions, but the invention provides for these same capabilities to apply to other 
signal formats in digital, analog, contact closure, entirely software, etc. or any combination. 

7.4.1 Gen ral control signal switching and m rging 

5 Referring to Figures 1-2, input signals directed to control routing and merging include the 

control outputs from the instrument entities 100 (including foot controllers), control signal 
extraction elements 128a, control signal processing elements 123, and control signal syn- 
thesis elements 129b. Still referring to Figures 1-2, output signals directed from control 
routing and merging include control inputs to instrument entities 100, all elements within 
10 the signal routing, processing, and synthesis entity 120 (probably excluding the power sup- 

ply 121), as well as any external lighting and/or special effects control systems. 

Using MIDI messages and conventions as a model, control signals may be carried through 
cables and subsystems in combinations of multiplexed formats (the sixteen MIDI channels 
plus the variety of message types) and space-division formats (multiple MIDI cables). In 
the MIDI context the invention provides for control signal routing at the MIDI port (i.e., 
MIDI cable) level, the MIDI channel level, and the message index (MIDI note numbers, 
MIDI Continuous Controller numbers, etc.) level. This same hierarchy of routing capabil- 
O ities would also apply to non-MIDI control signal equivalents. The invention also provides 

^ for the processing of control signals at any of these levels. 

\1 The MidiTemp MIDI processor products are by far the most comprehensive commercial 

ry products known at this writing; they provide full-capability port level and channel level 

=p routing but only very limited capabilities at the message index level. Further, the invention 

P provides for control switching and merging functions to preferably be an integrated com- 

=25 ponent within a larger-scale hardware and software construct rather than an off-the-shelf 

L module. 

fy To aid in using control signals throughout the system, the invention also provides for visual 

!■* indicators of control message value, such LED bar-graphs which may be accessed through 

jO^ control signal routing. 

U 7.4.2 Multi-channel Control Signal and Stored Program Handling and Organization 

The invention provides for a flexible control and configuration hierarchy for signal routing, 
processing, and synthesis entities. 

35 Figure 63 illustrates an example flexible control and configuration hierarchy for control 

signal and stored program handling and organization. In the Figure, all "program" entities 
(with the potential exception of the configuration program 6310) are stored programs 
which can be recalled up and swapped under the command of control signals (such as MIDI 
Program Change commands). Each stored program can in turn case the recall and/or swap- 

40 ping of other stored programs in accordance with those arrowed lines between pairs of pro- 

grams as shown in Figure 63, 

Referring to Figure 63, input control signals 6300.1 - 6300.n from instrument entities, foot 
controllers, etc. are applied to a control signal routing and control signal processing envi- 
ronment 63 01. The control signal routing and control signal processing environment 6301 
^ may internally be decomposed into separate control signal routing and control signal pro- 

cessing elements or instead be integrated together in a common realization (as is common 
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on many MIDI signal routing and handling products). The control signal routing and con- 
trol signal processing environment 6301 then distributes output control signals 6302.1 - 
6302. m throughout the rest of the system, specifically all classes of elements depicted in 
Figure 63, potentially including control aspects of itself. Among the elements receiving 
control signals is a configuration program 6310 which potentially provides simply a back- 
ground environment defining specific ports, safeguards, and any common control distribu- 
tion frameworks (point to point, broadcast, daisy-chain) should this be necessary. A variety 
of configuration programs 6310 may be made available for varying operational modes (for 
example stand-alone operation, ganged operation with one or more other signal routing, 
processing and synthesis entities, backup modes, diagnostic modes, etc. A more significant 
receptor of the output control signals 6302.1 - 6302.m is the master program 6320. This 
stored program can change the configuration of the control signal routing and control signal 
processing environment 6301 as well as the choice of any of the subsystem stored control 
programs 6331 - 6337 affiliated with sequencing, audio, lighting, etc. These control pro- 
grams in turn may change the configuration of the control signal routing and control signal 
processing environment 6301 (specifically 6331, 6332, 6334, 6335 are shown with this ca- 
pability in Figure 63) as well as the choice of any of the stored programs in the clusters of 
specific subsystem elements represented by 6342 - 6347 as well as the potentially external 
lighting subsystem element cluster 6348; specifically these stored program command paths 
are as indicated by the arrows in Figure 63. Further, the sequencer control program calls up 
and potentially immediately initiates real-time control signal sequences 6341 some of 
which may also change the configuration of the control signal routing and control signal 
processing environment 6301 as indicated by the arrowed line in Figure 63. 

It is understood that Figure 63 serves as an illustrative example and that the invention pro- 
vides for other organizational structures of this flavor and spirit. 

Control Signal Processing 

The invention provides for control signal processing to be included so as to add extensive 
valuable control capabilities. For convenience these control signal processing operations 
are described in terms of MIDI; the invention provides for these capabilities in other control 
signal formats as well. 

• Monodic operations: 

• intelligent harmony (note by note remapping, individually or in ranges of 
arbitrary size) 

• note-number to MIDI Continuous Controller values 

• note-velocity to MIDI Continuous Controller values 

• MIDI Continuous Controller values to note number messages 

• MIDI Continuous Controller value transformed by fixed scaling and offset 
values 

• MIDI Continuous Controller values (0-127) remapped to arbitrary 
mappings by point, by line segment, or by fitted curve segment 
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• MIDI Continuous Controller complementary value transformation (i.e., if 
received value is "x", transmitted value is "127-x") 

• message delay 

5 • message value threshold tests resulting in the issuances of new messages 

• message value threshold tests resulting in selected routing choices for the 
received message. 

• Polyadic operations: 

10 • multiplication of MIDI Continuous Controller values 

• scaling and offset of MIDI Continuous Controller values controlled by other 
MIDI Continuous Controller values 
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MIDI Continuous Controller to Note number and Note velocity 

• sequence detection in a received series of MIDI messages, potentially 
within a defined time window, resulting in a new issued message 



p 7.6 Control Signal Extraction 

I |q The invention provides for the extraction and derivation of control signals from audio and 

1 1 video signals as described below 

7.6.1 Audio signal to MIDI Note Event 

P The invention provides for the conversion of received audio signals into note events as is 

-25 standardly done in products such as the Roland GP-10, GM-70, and CP-40. The invention 

! = also provides for more advanced extractions and derivations as explained below. 

y 7.6.1.1 Envelope Tracking to MIDI 

J I The above conversions of received audio signals into note events as is standardly done in 

:j§ products such as the Roland GP- 1 0, GM-70, and CP-40 have been limited to channel allo- 

\2 cation, note number and note velocity. The invention provides for the real-time extraction 

of amplitude envelope information and its conversion to control signals. For example, the 
amplitude envelope may be used to control a signal processor or signal pan location. Be- 
cause the amplitude envelope falls off in a typically exponential way over time while most 
35 control structures expect linear variation, the invention provides for one or more possible 

warpings of the envelope signal, such as logarithm or piece-wise linear constructs. Further, 
the invention also provides for high-pass, band-pass emphasis/notching, and low-pass fil- 
tering prior to parameter extraction so as to limit unwanted influence of audio signal tran- 
sients at the initial execution of a vibrating element or audio synthesized note. 

^ 7.6.1,2 Control Signal Extract from Vibrating Element Overtones 

The use of pitch-detecting interfaces for converting the pitched vibrations of individual vi- 
brating elements into control signals for use with synthesizers or other musically-oriented 
signal processing has been in use for many years, particularly since shortly after the inven- 
45 tion of the MIDI standard for electronic instrument control. However, such pitch-detecting 

interfaces have derived only the fundamental frequency and overall amplitude of the 
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pitched vibrations of individual vibrating elements of an instrument. The use of filter banks 
for determining the energy in course frequency bands for the purposes of controlling mu- 
sically-oriented signal processing (i.e., the so-called vocoder") is also known. However, the 
practice of determining the scale-accurate pitches and amplitudes of individual overtones 
for the purposes of controlling synthesizers or other musically-oriented signal processing 
is currently not known. 

Current synthesizer interfaces (such the Boss GP-10 for guitars and the Zeta products for 
violins) typically only respond to the fundamental vibrating pitch and the overall ampli- 
tude. Further, amplitude responses in these current synthesizer interfaces typically only re- 
spond to the amplitude at the initial attack of a note and the event where the amplitude of 
the sustained vibration falls below a certain threshold. 

The invention provides for an expansion of traditional synthesizer control interfaces for vi- 
brating elements so as to respond to the pitches and amplitudes of higher-order overtone 
vibrations and issue control signals based on these. By expanding the response of tradition- 
al synthesizer interfaces for vibrating elements to include continuous time response to fun- 
damental and overtone amplitudes as well as pitches, far more expressive control over 
synthesized sound via tracking of vibrating elements can be obtained. For example, pluck- 
ing or bowing a string in varying locations can be used to control signal processing param- 
eters. 

Traditionally, synthesizer interfaces for vibrating elements capture pitch (based on funda- 
mental frequency of vibration) and amplitude, initially when a vibrating element is excited 
and in some cases as pitch and/or even amplitude changes dynamically. However, this can 
be expanded to include responses to various higher-order (non-fundamental) harmonics or 
other modes of vibration. It is important to note that such a feature can add tremendous con- 
trol over conventional synthesizer sound production in general situations where vibrating 
elements are used to control the synthesis of the sound; this is true somewhat in guitars, but 
much more so in wind and bowed instruments. In using a vibrating element feedback exci- 
tation arrangement for guitars, for example, this type of control signal extraction may be 
especially expressive as the feedback process can create widely-varying harmonic content 
when hands touch vibrating elements in feedback excitation or by varying the excitation 
feedback characteristics (via signal processing within the feedback loop). Because of the 
dynamic overtone characteristics of exciting vibrating elements in feedback loops, it is of 
interest to expand traditional synthesizer interfaces for vibrating elements to respond to the 
pitches and amplitudes of these higher-order overtone vibrations. The significant synergis- 
tic value of the combination of vibrating element excitation and overtone tracking control 
signal extraction are also recognized as part of the invention. 

There are various ways to accomplish such overtone tracking. In general, it is much easier 
for instruments whose elements vibrate at fixed pitches with a known overtone series. In 
these instruments, the overtone frequencies of a given vibrating element are also known in 
advance. Figure 64 shows an example method for the generation of control signals from 
fundamental and overtone information in a signal from a vibrating element of fixed known 
pitch. If the pitch is both fixed and known, the signal from each vibrating element can be 
filtered by a suite of band-pass filters 6402.1 - 6402 Jt, each separately tuned to the known 
frequency of an individual mode of vibration for that particular element. The output of each 
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filter can be fed to a dedicated amplitude follower 6403.1 - 6403 Jt. Each amplitude follow- 
er output can be used to create a separate parameter which can be assigned to an outgoing 
control signal 6405 via parameter mapping operation 6404. 

The invention provides for the combining and/or processing of fundamental and overtone 
information in creating yet other derived control signals. Figure 65 shows combining 
and/or processing of fundamental and overtone information obtained from a vibrating ele- 
ment signal prior to parameter assignment to control signals. As shown in Figure 65, the 
outputs of groups of amplitude detectors associated with a given vibrating element can be 
combined and/or processed 6406 before mapping to final parameters prior to control signal 
assignment. For example, different weighted sums can be used to control the amplitude of 
a synthesized signal (say a uniform averaging, or sum-of-squares averaging) than would be 
used to control the cut-off frequency of a subtractive filter (here, weighting the higher 
modes of vibration more strongly would make the synthesis mimic the vibrating element's 
harmonic balance; weighting the lower more strongly would make the synthesis comple- 
ment the vibrating element's harmonic balance, etc.). 

In the case where the vibrating elements do not vibrate at a fixed pitch but still obey a 
known overtone relationship, a slightly more involved version of the same mechanism can 
*5 also be used. Note that such an implementation is hardly limited to feedback systems and 

gjp could be used in general guitar and violin synthesizer interfaces for new depths of perfor- 

jl mance control. In addition, because variation in overtone series dynamics is an essential 

m factor in singing and in percussion instruments, such a technology opens important new 

=p doors for synthesizer overtone-nuance tracking for voice and percussion instruments. In 

(3 singing in particular, the relative amplitudes of the first three harmonics (largely the first 

§5 two, actually) determine the choice of sung vowel; as a result, this technology allows syn- 

thesizers to track the formants of vowel production in the human voice. 

I j Figure 66 shows an example implementation of an adaptive method for tracking overtones 

M for a variable-pitch vibrating element with known overtone series. The method is largely 

=0 the same as the fixed-pitch case, but with some added steps. The additional steps employed 

3P are to first use a traditional pitch detector 6407 (as used with conventional MIDI guitar/vi- 

olin/voice interfaces) to determine the fundamental pitch, then use this pitch information 
plus an overtone series model of the vibrating element 6400 to position the frequencies of 
the individual band-pass filters 6402.1 - 6402.h and amplitude followers 6403.1 - 6403. h. 

35 In a preferred implementation of this approach, the detected pitch information provided by 

the pitch detector 6407 is fed to a model-based overtone series calculator 608. The model- 
based overtone series calculator 155 generates the control signals required to individually 
center each of the plurality of band-pass filters 6402. 1 - 6402. h. The model-based overtone 
series calculator 6408 is also used to generate overtone frequency information for use in 

40 any combining or processing of the extracted overtone amplitude information and in the pa- 

rameter mapping 6404 to final output control signals. 

7.6.2 Pluck direction to MIDI 

The invention provides for the extraction of plucking direction (as on an instrument string) 
45 of arbitrary vibrating element and creating a control signal from it. Core technologies for 

detecting pluck direction typically include separate analysis of the signals from a 2 - coil 
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humbuck pickup and have been implemented in products by Biax and Passaic. Passaic also 
implemented a method for deriving a control parameter from where a string was plucked 
between the bridge and the neck. The invention provides for these extraction functions to 
be included in the available control extraction capabilities. 

5 

7.6.3 Video Motion and Feature Extraction 

The invention provides for the extraction of parameters from provided video signals as de- 
cribed earlier and creating control signals from them. Methods for implementing this have 
been described earlier, including simple timing tests and video frame grabs analyzed by 
10 dedicated systems or personal computer software. The invention also provides for imple- 

mentations using emerging motion tracking and image decomposition methodologies un- 
der development for widespread adoption in digital video compression standards such as ' 
MPEG-4 (see for example [Hara; Bormans]. 

15 7.6.4 Control Signal Pattern Recognition 

The invention provides for the recognition of control signal patterns. Since the result is yet 
another control signal, this has been treated earlier in the context of control signal process- 
es ing ' 

£0 7.7 Dynamic Control Signal Synthesis 

!^ The invention provides for the synthesis of dynamic control signals such as low-frequency 

= 2 sweep oscillators, particularly those whose parameters may be controlled in real-time by 

i5 other control signals. Since an envelope generator trigger is also a control signal, the gen- 

:ig eration of control signal envelopes and slews are also included in this category and are pro- 

= vided for by the invention. 

iTj 7.7.1 MIDI-controlled low-frequency control oscillators ensembles 

*~ Low-frequency sweep oscillators, or LFOs, have roles throughout the invention and have 

£Q been discussed earlier. In some types of functions implemented by specific elements, such 

J J as chorus and flangers, the LFO may be hard associated with the element. The invention 

provides for this as well as the remote positioning of the LFO function outside the element 
in the case where several elements may be coordinated with the same LFO. In other types 
of functions, such as location modulation, it may be best to control existing elements such 
as mixers with control signals from external LFOs. 

The invention provides for a plurality of control signal LFOs to be available. The LFOs 
may be part of a comprehensive system or a separate module which can be manufactured 
and sold for other uses; such a product would be naturally served by at least MIDI output 
and input, but may also include at least one analog input and/or output. The LFOs provided 

40 for by the invention include multiple phase output capabilities as well as selections of a va- 

riety of waveforms, frequency settings, amplitude settings and offset settings, all of which 
may be varied in real-time by yet other control signals. Further, the invention provides for 
these parameters to be available under selectable stored program control which may be cho- 
sen by control signals. Finally, the invention provides for global effects across groups of 

45 LFOs, such as timing slew of parameter changes, global scaling, global offsets, etc. These 

may also include more complex organizations such as may be require for two-dimensional 
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and three-dimensional location modulation and the custom construction or sampling of 
LFO waveforms. 

Figure 67 illustrates an example approach wherein a plurality of LFOs with features as pre- 
scribed by the invention may be implemented. Program data for M different stored pro- 
grams may be stored in M data structures 6700.1 - 6700.M; these may include specific LFO 
parameters 6701.1.1 - 6701.M.N for N LFOs. The data structures may also include global 
information 6702.1 - 6702M pertaining to groups of LFOs. A received control signal 
and/or panel control may be used to designate the selected program 6703 which is recalled 
and implemented 6704 as a set of default vaules subject to real-time change by other control 
signals. The LFO data vector structure for each LFO includes a source of select information 
6711 for choosing whether a given output operates as an independent LFO or as a slave to 
another LFO specified here (and in so doing becomes a multiple phase output for the cho- 
sen source oscillator). Another part 6712 of the LFO data vector provides a frequency set- 
ting if the LFO is independent and a phase setting if slaved; should integer-ratio phase- 
locking be implmeneted, this part of the data structure may be reorganized to include rela- 
tive frequency and phase settings with respect to the selected master LFO. Another part 
6713 of the LFO data vector provides selected waveform information, including reference 
to any user sampled LFO waveforms. Another two parts 6714, 6715 of the LFO data vector 
provide respectively amplitude scaling and offset settings. Additional information, such as 
the outgoing MIDI channel and MIDI Continuous Controller number to be used and what 
incoming MIDI Continuous Controller messages on what MIDI channel are used to control 
the aforementioned LFO settings. The global part of the data 6731, 6732 provides global 
information for specific settings of global amplitude, offset, parameter, time slew, etc. per- 
taining to specific groups of LFOs. The selected information is presented to the LFO engine 
for execution, The LFO engine in the example implements an independent LFO by dividing 
6751 a system clock signal 6741 by a number determined by the frequency setting 6712. 
The divided clock signal runs a counter 6752 (here a 128 step counter is illustrated, al- 
though higher resolution may be desired). Should the LFO be instead designated, via the 
information 6711, to be slave to another LFO, the counter of that master LFO is accessed 
6753 and the phase offset information 6712 is used to create a phase offset value which is 
provided together with the accessed counter value to an modular adder 6755 which produc- 
es the resulting phase-shifted version of the master LFO count value. The resulting count- 
ing sequence produced by this section 6750 is used as the address for a waveform lookup 
table 6756 and/or algorithm; the resulting generated periodic signal is then scaled 6757 ac- 
cording to 6714 and offset in amplitude 6758 by information 6715. The resulting wave- 
forms can be post-processed to provide global amplitude and offset features, or, alternative, 
mathematical transformations 6731 may be provided on the information 6714, 6715 before 
executing it in the LFO engine in elements 6757, 6758. 

Controlled slews, ramp, and envelope generator elements 

The invention provides for slew limiters, ramp generators, and envelope generators whose 
trigger and various parameters may be varied in real-time by control signals. Slew limiters 
limit the rate of change of a control signal to a maximal range which may be set as a pa- 
rameter and advantageously varied by control signals. Ramp generators are simplified en- 
velope generators triggered by control signals which ramp between two or more discrete 
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values or the entire control signal range and do so according selected types of dynamics 
(linear over time, exponential over time, etc.); the parameters here may be set and advan- 
tageously varied by control signals. Envelope generators offer more complex transient 
waveforms, typically including at least attack, decay, sustain, and release; more complex 
5 envelope features including more breakpoints, delays, and segment curve shapes may also 

be provided. The parameters here may be set and advantageously varied by control signals. 

7.8 Lighting Effects and Video Display 

The invention provides for extensive control of lighting via control signals. Some aspects 
of lighting as provided for by the invention are described in the sub-sections below. 

7.8.1 Light Types 

The invention provides for extensive control of lighting via control signals. Some aspects 
^ of lighting as provided for by the invention are described in the sub-sections below. 

7.8.1.1 Traditional fixed 

The invention provides for traditional fixed lighting arrangements as shown in Figure 68. 
O These may include any one or more of overhead lights 6802, far-throw lights 6807, foot 

lights 6803, floor lights aimed upward or at angles 6804, backlights 6806 behind equipment 
and risers 6805 and backdrops 6801. 

7.8.1.2 Movable 

The invention provides for movable lighting controlled in real-time via control signals. 
Such lighting can be implemented by attaching lights to motorized pan/tilt heads as used 
for video cameras. 
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7.8.1.3 Instrument lighting 

U- The invention provides for lighting on instrument entities which may be operated via con- 

=0 trol signals. Figure 69 shows examples of lighting for a guitar 6900 from the bridge 6901, 

ffO neck 6902, above 6904 and below 6903 the picking area, all aimed at illumination effects 

for the hands. Also shown are lights aimed at the audience in the pickup areas 6905, 6906 
and fret areas 6907, any of which may be aggregate as in 6905, 6906 or split out separately 
for each string as in 6907. 

2$ 7.8.1.4 Light sculptures 

The invention provides for light sculptures under control of control signals. Figure 70 
shows rotating speaker emulation light sculptures. In one implementation a rotating reflec- 
tive beacon 701 1 reflects gathered light from a bulb 7012 and projects it on to a translucent 
concealing cover 7013 attached to, in this case, a pyramid frame. An alternate arrangement 

40 where the translucent outer structure 7023 itself rotates is also shown about a fixed light 

bulb 7022 whose cable 7026 fits through the bearings 7024, 7025 of a rotating turntable 
7021,7024 driven by geared 7027, 7028 motor arrangement. The mechanism just described 
is exploded for explanation and in fact may be readily collapsed by standard means so as 
to permit two rotating pyramids 703 1 7032 to be stacked in transparent cubes 7033, 7034. 

45 the motor speed and direction can be controlled by control signals and arranged to operate 

in synchronization with rotating speaker simulations in audio signal processing elements/ 
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Figure 71 shows light pyramids 7100 made of similar elements 7101-7106, typically with- 
out reflectors or directional bulbs, which may be arranged in arrays 7108. Also shown are 
light column arrays suggestive of organ pipes 71 18 or instrument strings 7119. These light 
columns may be built with standard lamps and reflectors 71 12, 71 14, 71 15, colored gels 
5 7111 and light transmitting and scattering rods 7110. 

Also provided for by the invention are controlled ionize gas turbulence sculptures; these 
may be used with or without associated video ameras. 

7.8.2 General Lighting Control 

^ The invention provides for lights to be used in scene change modes or modulated by control 

signals according to: 

• animation sequences and subsequence events 

• instrument activity 

• timbre qualities 

Special Instrument lighting effects include: 

"z. • audience shock events 

=o 

BP • animation sequences 

\^ 

\ v • string activity, note following, orchestration following 
% 7.8.3 Video Signal Routing 

g Referring to Figures 1-2, input signals directed to video routing include the video outputs 

i from the instrument entities 100, video signal processing elements 127, video signal syn- 

P thesis elements 129a, and external video feeds from, for example, miscellaneous stage 

m cameras, VCRs, etc. Still referring to Figures 1-2, output signals directed from video rout- 

'*Z ing include video inputs to instrument entities 100, video signal processing 127, control sig- 

J|q nal extraction 128a, and overall video outputs to one or more of any displays, projectors, 

12 and/or recording facilities. 

7.8.4 Video Signal Processing 

Video signal processing as provided for by the invention would include overlays, wipes, 
35 fades, blends, solarizations, geometric warping, etc, as much as possible under the control 

of control signals. Interesting effects provided for by the invention include the switching, 
wiping, blending, fading, warping, etc. of various video signals for display in performance 
and /or recording under the control of instrument note and amplitude envelope signals. 

7.8.5 Video Display 

40 

Figure 72 shows how the invention provides for video projection to be used to shine down 
7201 on the stage 7200, shine horizontally 7202 on to performers or backdrops. The inven- 
tion also provides for video projectors to be shined on the audience. The invention also pro- 
vides for The invention provides for movable cameras controlled in real-time via control 
45 signals using motorized pan/tilt heads as well as motorized zoom/focus lenses. ) 
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7.8.6 Vide Signal Synth sis 

The invention provides for video signal synthesis would include real-time generation of 
text message screens, text overlays, vector and raster graphic drawings, vector and raster 
graphic overlays, and animations affiliated with numerical dynamics simulation. The in- 
vention also provides for pre-stored video frames, playback of video clips, and playback of 
prestored vector and raster graphics animations. The invention provides for these to be con- 
trolled by standardized control signals, such as MIDI, and as such would typically involve 
both stored program control and parameterized control. These functions may be realized 
with a conventional personal computer fitted with video card and MIDI interface as well as 
by dedicated hardware. 
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8 Example Envisioned Applications 

j 5 A few example envisioned applications of the invention are now provided. 

8.1 Add-on Modules for existing instruments 

q This gives rise to a whole new marketplace for new instruments, instrument retrofit kits, 

\Q and music signal processor units which can interact with external amplifiers, signal pro- 

20 cessing, and MIDI synthesizer units. 

^ 8.2 Creation of enhanced electronic vibrating element instruments 

p With the first technique described within this patent, the more traditional acoustically-ex- 

cited "controlled feedback" effects can easily be obtained, via electromagnetic excitation, 
with standard parts. Specialization of the parts can provide additional features. The tech- 
O nique can also be applied to any instrument where sound is produce by vibrating ferromag- 

\* netic material, e.g., African mbiras, violins, xylophones, etc. 

bp With the second technique described in this patent, conventional signal processing can be 

CD used on each string signal to create "generalized pedal steel guitars," multi-modal Indian 

!=== sitars (where drone and sympathetic strings can be electronically retuned while playing), 

spatially animated string sounds within a stereophonic or spatial sound field, and mixed 
timbre instruments where different signal processing methods are applied to each string. 
The technique can also be applied to any instrument where vibration of individual sound- 
35 producing elements can be electronically captured by isolated transducers (electromagnet- 

ic, optical, Hall-effect, etc.), such as nylon-stringed instruments, marimbas, African mbi- 
ras, violins, etc. 

By combining these two new techniques with appropriate signal processing, a very power- 
ful environment for multi-stringed electronic instruments can be created. Individual strings 
can be singled out for feedback operation while others operate without feedback, and all 
strings can be electronically pitch-shifted as needed in a performance. The results allow a 
performer a greater degree of polyphonic control, using mechanical (neck, frets, fingers, 
picks, movable tailpieces, pedal tuning changers, etc.) or electronic means for both string 
excitation and pitch control, with individual string outputs available for synthesizer inter- 
faces. 
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Any to all of the above can be built into an individual instrument. Alternatively, an instru- 
ment interface can be created and most signal processing can be remotely located from the 
instrument, connecting to it via this interface. If this interface is standardized across multi- 
ple instruments, then common signal processing equipment environment can be used 
across a wide variety of instruments (metal-stringed and nylon-stringed guitars, basses, vi- 
olins, steel guitars, sitars, mbiras, etc.). This gives rise to a whole new marketplace for new 
instruments, instrument retrofit kits, and music signal processor units which can interact 
with external amplifiers, signal processing, and MIDI synthesizer units. 



All publications and patent applications mentioned in this specification are herein incorpo- 
rated by reference to the same extent as if each individual publication or patent application 
was specifically and individually indicated to be incorporated by reference. The invention 
now being fully described, it will be apparent to one of ordinary skill in the art that many 
changes and modifications can be made thereto without departing from its spirit or scope. 
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